Portal:Toolforge/Admin/Maintenance

This page contains information for common maintenance tasks we administrator do in Toolforge.

Is the Toolforge-specific version of the CloudVPS admin maintenance conterpart wikipage.

Admin scripts

This section contains documentation on several Toolforge-specific admin scripts we have.

exec-manage

This script helps manage Grid Engine exec nodes by allowing for depool, repool, and check status, given an exec node hostname.

Usage is:

• exec-manage [status|depool|repool] exec_host_name
• exec-manage [list]

Example:

root@tools-sge-master-01:~# exec-manage status tools-sgeexec-1001.tools.eqiad.wmflabs
[..]

wmcs-k8s-get-cert

This script is for the new Toolforge kubernetes cluster.

It creates a x509 cert for a given ServiceAccount that can be used to auth against the k8s API. It must be executed from a control plane node.

user@tools-k8s-control-3:~$ sudo wmcs-k8s-get-cert -help
INFO: Usage of this script:
      /usr/local/sbin/wmcs-k8s-get-cert -h/--help   	- show help and exit
      /usr/local/sbin/wmcs-k8s-get-cert <svcname>   	- generate a x509 TLS cert from the kubernetes API
      /usr/local/sbin/wmcs-k8s-get-cert <svcname> -v	- same, but in verbose mode

Example usage:

user@tools-k8s-control-3:~ $ sudo -i wmcs-k8s-get-cert myserviceaccount
/tmp/tmp.JYFsVzDnX4/server-cert.pem
/tmp/tmp.JYFsVzDnX4/server-key.pem

NOTE: x509 certs expire in 1 year usually. See also Portal:Toolforge/Admin/Certificates_in_k8s

wmcs-k8s-enable-cluster-monitor

This is a script for the new Toolforge Kubernetes cluster, deployed to the control plane nodes.

It will give a specified tool account a service account named $tool-obs that will allow a pod running as it to have "view" access to the entire cluster. This is for building tools similar to Openstack Browser for Kubernetes.

user@tools-k8s-control-3:~$ sudo -i  # You'll want full root for k8s credentials
root@tools-k8s-control-3:~# wmcs-enable-cluster-monitor <tool-name>

If that runs successfully, you'll want to run the tool as that service account using a script like k8s_webservice.sh instead of using webservice.

Important point about using a service account: it must use the credentials mounted at /var/secrets/kubernetes.io/serviceaccount in the pod and not $HOME/.kube/config or you'll get the tool permissions instead of the $tool-obs permissions. This is largely because of invoking the pod presets to get the $HOME variable set to /data/project/$tool and the NFS mounted.

wmcs-k8s-secret-for-cert

This script is for the new Toolforge kubernetes cluster.

It creates a k8s secret to hold x509 cert for a given ServiceAccount which you can use inside a pod to auth to the k8s API server.

user@tools-k8s-control-3:~$ sudo wmcs-k8s-secret-for-cert -h
INFO: this script creates a k8s secret to hold a x509 cert for a
      given service account (pub/priv) which you can then use inside
      a pod, mounting the secret as a volume. Usage:

  /usr/local/sbin/wmcs-k8s-secret-for-cert [-n <namespace>] -s <secretname> -a <svcname> [-v] | -h

    -n namespace    the namespace in which the secret will be created (uses default if not specified).
    -s secretname   name of the secret being created. This is the name you use to mount it later on.
    -a svcname      service name, doing RBAC auth for this service account.
    -v              verbose mode.

    -h              show help and exit.

Example usage:

user@tools-k8s-control-3:~ $ sudo -i wmcs-k8s-secret-for-cert -n kube-system -s mysecret -a myserviceaccount
secret/mysecret created

This script is calling internally other admin script, wmcs-k8s-get-cert.

When the secret is created, it can be used in a pod like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  namespace: my-namespace
spec:
  template:
    metadata:
      name: my-whatever
    spec:
      serviceAccountName: my-serviceaccount
      volumes:
      - name: my-secret-cert
        secret:
          secretName: my-secret-cert
      containers:
      - name: my-pod
        image: whatever
        args:
          - --tls-cert-file=/etc/certs/cert.pem
          - --tls-private-key-file=/etc/certs/key.pem
        volumeMounts:
        - name: my-secret-cert
          mountPath: /etc/certs
          readOnly: true
       [..]

NOTE: X509 certs expire in 1 year. See also Portal:Toolforge/Admin/Certificates_in_k8s

Other scripts

Other scripts you may find interesting.

wmcs-package-build

This script is used to build and publish packages into aptly. More information in the packaging page.

user@laptop:~$ modules/toolforge/files/wmcs-package-build.py --help
usage: wmcs-package-build.py [-h] --git-repo GIT_REPO [--git-branch GIT_BRANCH]
                             [--build-dist BUILD_DIST] [--build-host BUILD_HOST] [-a APTLY_DIST]
                             [--aptly-host APTLY_HOST] [-b] [-d]

Utility to build and upload a .deb package to aptly

optional arguments:
  -h, --help            show this help message and exit
  --git-repo GIT_REPO   git repository URL with the source pkg. This script will do a fresh git clone
                        of that repo. Typical value is something like:
                        https://gerrit.wikimedia.org/r/operations/software/tools-webservice
  --git-branch GIT_BRANCH
                        git branch to use to build the package from. Defaults to "master"
  --build-dist BUILD_DIST
                        target distribution when building the package with sbuild. Defaults to
                        "stretch"
  --build-host BUILD_HOST
                        package build host. Typically a VM in CloudVPS with
                        role::wmcs::toolforge::package_builder. Defaults to "tools-package-
                        builder-02.tools.eqiad.wmflabs"
  -a APTLY_DIST, --aptly-dist APTLY_DIST
                        target distribution in aptly. The resulting deb package will be uploaded to
                        this distribution and then the repository will be published. Can be specified
                        multiple times for multiple target distributions. If this argument is not
                        provided, no aptly operations will be done. Example: -a stretch-tools -a
                        stretch-toolsbeta
  --aptly-host APTLY_HOST
                        aptly server host. Typically a VM in CloudVPS with
                        role::wmcs::toolforge::services. Defaults to "tools-sge-
                        services-03.tools.eqiad.wmflabs"
  -b, --no-backup       If this option is present, this script won't backup aptly data over NFS
  -d, --dry-run         Dry run: only show what this script would do, but don't do it for real

This script is supposed to run from your laptop, and it will start SSH connections to the different servers involved in building and publishing a deb package.

The --help output should have enough information to know what every option does.

You are encouraged to use the --dry-run first to know what the script would do (the actual SSH commands).