Portal:Toolforge/Admin/Maintenance

This page contains information for common maintenance tasks we administrator do in Toolforge.

Is the Toolforge-specific version of the CloudVPS admin maintenance conterpart wikipage.

Admin scripts

This section contains documentation on several Toolforge-specific admin scripts we have.

exec-manage

This script helps manage Grid Engine exec nodes by allowing for depool, repool, and check status, given an exec node hostname.

Usage is:

• exec-manage [status|depool|repool] exec_host_name
• exec-manage [list]

Example:

root@tools-sge-master-01:~# exec-manage status tools-sgeexec-1001.tools.eqiad.wmflabs
[..]

wmcs-k8s-get-cert

This script is for the new Toolforge kubernetes cluster.

It creates a x509 cert for a given ServiceAccount that can be used to auth against the k8s API. It must be executed from a control plane node.

user@tools-k8s-control-3:~$ sudo wmcs-k8s-get-cert -help
INFO: Usage of this script:
      /usr/local/sbin/wmcs-k8s-get-cert -h/--help   	- show help and exit
      /usr/local/sbin/wmcs-k8s-get-cert <svcname>   	- generate a x509 TLS cert from the kubernetes API
      /usr/local/sbin/wmcs-k8s-get-cert <svcname> -v	- same, but in verbose mode

Example usage:

user@tools-k8s-control-3:~ $ sudo -i wmcs-k8s-get-cert myserviceaccount
/tmp/tmp.JYFsVzDnX4/server-cert.pem
/tmp/tmp.JYFsVzDnX4/server-key.pem

NOTE: x509 certs expire in 1 year usually. See also Portal:Toolforge/Admin/Certificates_in_k8s

wmcs-k8s-enable-cluster-monitor

This is a script for the new Toolforge Kubernetes cluster, deployed to the control plane nodes.

It will give a specified tool account a service account named $tool-obs that will allow a pod running as it to have "view" access to the entire cluster. This is for building tools similar to Openstack Browser for Kubernetes.

user@tools-k8s-control-3:~$ sudo -i  # You'll want full root for k8s credentials
root@tools-k8s-control-3:~# wmcs-enable-cluster-monitor <tool-name>

If that runs successfully, you'll want to run the tool as that service account using a script like k8s_webservice.sh instead of using webservice.

Important point about using a service account: it must use the credentials mounted at /var/secrets/kubernetes.io/serviceaccount in the pod and not $HOME/.kube/config or you'll get the tool permissions instead of the $tool-obs permissions. This is largely because of invoking the pod presets to get the $HOME variable set to /data/project/$tool and the NFS mounted.

wmcs-k8s-secret-for-cert

This script is for the new Toolforge kubernetes cluster.

It creates a k8s secret to hold x509 cert for a given ServiceAccount which you can use inside a pod to auth to the k8s API server.

user@tools-k8s-control-3:~$ sudo wmcs-k8s-secret-for-cert -h
INFO: this script creates a k8s secret to hold a x509 cert for a
      given service account (pub/priv) which you can then use inside
      a pod, mounting the secret as a volume. Usage:

  /usr/local/sbin/wmcs-k8s-secret-for-cert [-n <namespace>] -s <secretname> -a <svcname> [-v] | -h

    -n namespace    the namespace in which the secret will be created (uses default if not specified).
    -s secretname   name of the secret being created. This is the name you use to mount it later on.
    -a svcname      service name, doing RBAC auth for this service account.
    -v              verbose mode.

    -h              show help and exit.

Example usage:

user@tools-k8s-control-3:~ $ sudo -i wmcs-k8s-secret-for-cert -n kube-system -s mysecret -a myserviceaccount
secret/mysecret created

This script is calling internally other admin script, wmcs-k8s-get-cert.

When the secret is created, it can be used in a pod like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  namespace: my-namespace
spec:
  template:
    metadata:
      name: my-whatever
    spec:
      serviceAccountName: my-serviceaccount
      volumes:
      - name: my-secret-cert
        secret:
          secretName: my-secret-cert
      containers:
      - name: my-pod
        image: whatever
        args:
          - --tls-cert-file=/etc/certs/cert.pem
          - --tls-private-key-file=/etc/certs/key.pem
        volumeMounts:
        - name: my-secret-cert
          mountPath: /etc/certs
          readOnly: true
       [..]

NOTE: X509 certs expire in 1 year. See also Portal:Toolforge/Admin/Certificates_in_k8s