Help:Toolforge/Kubernetes: Difference between revisions

Overview

Kubernetes (often abbreviated k8s) is a platform for running containers. It is used in Toolforge to isolate Tools from each other and allow distributing Tools across a pool of servers.

Kubernetes webservices

The Toolforge webservice command has a --backend=kubernetes mode that will start, stop, and restart containers designed to run web services for various languages. See our Webservice help for more details.

Kubernetes cronjobs

It is possible to run cron jobs on Kubernetes (see upstream documentation for a full description).

Example cronjob.yaml

Wikiloveslove is a Python 3.7 bot that runs in a Kubernetes deployment. The cronjobs.yaml file that it uses to tell Kubernetes how to start and schedule the bot is reproduced below.

After creating the cronjob you can create a test job with kubectl create job --from=cronjob/CRONJOB-NAME test to immediately trigger the cronjob and then access the logs as usual with kubectl logs job/test -f to debug.

/data/project/wikiloveslove/cronjobs.yaml (copied 2020-02-01)

The following content has been placed in a collapsed box for improved usability.

---
# NOTE: this cronjob works with the "toolforge" Kubernetes cluster, and not the legacy "default" cluster.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: list-images
  labels:
    name: wikiloveslove.listimages
    # The toolforge=tool label will cause $HOME and other paths to be mounted from Toolforge
    toolforge: tool
spec:
  schedule: "28 * * 2 *"
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            toolforge: tool
        spec:
          containers:
          - name: bot
            workingDir: /data/project/wikiloveslove
            image: docker-registry.tools.wmflabs.org/toolforge-python37-sssd-base:latest
            args:
            - /bin/sh
            - -c
            - /data/project/wikiloveslove/list_images.sh
            env:
            - name: PYWIKIBOT_DIR
              value: /data/project/wikiloveslove
            - name: HOME
              value: /data/project/wikiloveslove
          restartPolicy: OnFailure

The above content has been placed in a collapsed box for improved usability.

Kubernetes continuous jobs

The basic unit of managing execution on a Kubernetes cluster is called a "deployment". Each deployment is described with a YAML configuration file which describes the container images to be started ("pods" in the Kubernetes terminology) and commands to be run inside them after the container is initialized. A deployment also specifies where the pods run and what external resources are connected to them. The upstream documentation is comprehensive.

Example deployment.yaml

Stashbot is a Python 3.7 irc bot that runs in a Kubernetes deployment. The deployment.yaml file that it uses to tell Kubernetes how to start the bot is reproduced below. This deployment is launched using a stashbot.sh wrapper script which runs kubectl create --validate=true -f /data/project/stashbot/etc/deployment.yaml.

/data/project/stashbot/etc/deployment.yaml (copied 2020-01-03)

The following content has been placed in a collapsed box for improved usability.

---
# NOTE: this deployment works with the "toolforge" Kubernetes cluster, and not the legacy "default" cluster.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stashbot.bot
  namespace: tool-stashbot
  labels:
    name: stashbot.bot
    # The toolforge=tool label will cause $HOME and other paths to be mounted from Toolforge
    toolforge: tool
spec:
  replicas: 1
  selector:
    matchLabels:
      name: stashbot.bot
      toolforge: tool
  template:
    metadata:
      labels:
        name: stashbot.bot
        toolforge: tool
    spec:
      containers:
        - name: bot
          image: docker-registry.tools.wmflabs.org/toolforge-python37-sssd-base:latest
          command: [ "/data/project/stashbot/bin/stashbot.sh", "run" ]
          workingDir: /data/project/stashbot
          env:
            - name: HOME
              value: /data/project/stashbot
          imagePullPolicy: Always

The above content has been placed in a collapsed box for improved usability.

This deployment:

Uses the 'tool-stashbot' namespace that the tool is authorized to control
Creates a container using the 'latest' version of the 'docker-registry.tools.wmflabs.org/toolforge-python37-sssd-base' Docker image.
Runs the command /data/project/stashbot/bin/stashbot.sh run when the container starts.
Mounts the /data/project/stashbot/ NFS directory as /data/project/stashbot/ inside the container.

Monitoring your jobs

You can see which jobs you have running with kubectl get pods. Using the name of the pod, you can see the logs with kubectl logs <pod-name>.

To restart a failing pod, use kubectl delete <pod-name>. If you need to kill it entirely, find the deployment name with kubectl get deployment, and delete it with kubectl delete deployment <deployment-name>.

Namespaces

Each tool has been granted control of a Kubernetes "namespace". Your tool can only create and control objects in its namespace. A tool's namespace is the same as the tool's name (e.g. admin, stashbot, hay, etc).

Container images

The Toolforge Kubernetes cluster is restricted to loading Docker images published at docker-registry.tools.wmflabs.org (see Portal:Toolforge/Admin/Kubernetes#Docker Images for more information). These images are built using the Dockerfiles in the operations/docker-images/toollabs-images git repository.

Available container types

The webservice command has an optional type argument that allows you to choose which Docker container to run your Tool in.

Currently provided types:

golang (go v1.11.5; deprecated)
golang111 (go v1.11.6)
jdk11 (openjdk 11.0.5)
jdk8 (openjdk 1.8.0_232; deprecated)
node10 (nodejs v10.15.2)
nodejs (nodejs v6.11.0; deprecated)
php5.6 (PHP 5.6.33; deprecated)
php7.2 (PHP 7.2.24; deprecated)
php7.3 (PHP 7.3.11)
python (Python 3.4.2; deprecated)
python2 (Python 2.7.9; deprecated)
python3.5 (Python 3.5.3; deprecated)
python3.7 (Python 3.7.3)
ruby2 (Ruby 2.1.5p273; deprecated)
ruby25 (Ruby 2.5.5p157)
tcl (TCL 8.6)

For example to start a webservice using a php7.3 container, run:

webservice --backend=kubernetes php7.3 start

A complete list of images is available from the docker-registry tool which provides a pretty frontend for browsing the Docker registry catalog.

As of Feb 2018, we don't support mixed runtime containers. This may change in the future. Also, we don't support "bring your own container" on our kubernetes (yet!). And there is no mechanism for a user to install system packages inside of a container.

PHP

PHP uses lighttpd as a webserver, and looks for files in ~/public_html/.

PHP versions & packages

There are three versions of PHP available, PHP 7.3 (on Debian Buster), PHP 7.2 (on Debian Stretch), and the legacy PHP 5.6 (on Debian Jessie).

You can view the installed PHP extensions on the phpinfo tool. This should match the PHP related packages installed on GridEngine exec nodes. Additional packages can be added on request by creating a Phabricator task tagged with #toolforge-software. Software that is not packaged by Debian upstream is less likely to be added due to security and maintenance concerns.

PHP Upgrade

To upgrade from PHP 5.6 to PHP 7.3, run the following two commands:

$ webservice stop
$ webservice --backend=kubernetes php7.3 start

To switch back:

$ webservice stop
$ webservice --backend=kubernetes php5.6 start

Running Locally

You may run the container locally by executing a command like this:

$ docker run --name toolforge -p 8888:80 -v "${PWD}:/var/www/html:cached" -d docker-registry.tools.wmflabs.org/toollabs-php-web sh -c "lighty-enable-mod fastcgi-php && lighttpd -D -f /etc/lighttpd/lighttpd.conf"

Then the tool will be available at http://localhost:8888 Use docker-registry.tools.wmflabs.org/toollabs-php72-web for the php7.2 server.

Node.js

The container images for Node.js, such as docker-registry.tools.wmflabs.org/toollabs-nodejs-base:latest currently come with a current version of Node.js LTS from Wikimedia APT (as of September 2018, this is Node.js 6). This is the same version used by Wikimedia Foundation in production and for continuous integration.

Broken npm

Given npm is not suitable for use in Wikimedia production, the version of Node.js provided by Wikimedia APT is compiled without npm. (Unlike the official Node.js distribution.) And because there is no use for npm in Wikimedia production, there is no "npm" Debian package maintained in Wikimedia APT. The result is that the only "npm" Debian package available is the one from upstream Debian, which is npm 1.4 which was originally bundled in 2014 with Node 0.10 (debian/npm, debian/nodejs). This version is EOL and is incompatible with most packages on the npmjs.org registry. To update it within your container, follow these steps

# Step 1: Start a shell in your Node.js pod (see "Shell" section below)
tool@tools-login$ kubectl exec -it podname-123-aaa -- /bin/bash

# Step 2: Create $HOME/bin and ensure it is in your PATH
podname:/data/project/tool$ mkdir bin/
podname:/data/project/tool$ export PATH="${HOME}/bin:${PATH}"
# To avoid having to re-export PATH every time you use your tool, add the export command to your .bashrc file!

# Step 3: Use npm to install 'npm'
podname:/data/project/tool$ npm install npm
....
# This installs the current version of npm at node_modules/.bin/npm

# Step 4: Create a symlink in $HOME/bin
podname:/data/project/tool$ ln -s $HOME/node_modules/.bin/npm $HOME/bin/npm
# Close the shell and create a new shell (to initialise PATH)
podname:/data/project/tool$ exit
tool@tools-login$ kubectl exec -it podname-123-aaa -- /bin/bash
podname:/data/project/tool$

# Step 5: Verify that you now use a current npm instead of npm 1.4
podname:/data/project/tool$ npm --version
6.4.1

Troubleshooting

"failed to create new OS thread" from kubectl

If kubectl get pods or a similar command fails with the error message runtime: failed to create new OS thread (have 12 already; errno=11), use GOMAXPROCS=1 kubectl ... to reduce the number of resources that kubectl requests from the operating system.

Get a shell inside a running Pod

Kubectl can be used to open a shell inside a running Pod: $ kubectl exec -it $NAME_OF_POD -- /bin/bash

See Get a Shell to a Running Container at kubernetes.io/docs for more information.

[== Communication and support ==

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia Movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support

Chat in real time in the IRC channel #wikimedia-cloud ^connect, the bridged Telegram group, or the bridged Mattermost channel
Discuss via email after you subscribed to the cloud@ mailing list

Receive mail announcements about critical changes

Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)

Track work tasks and report bugs

Use the Phabricator workboard #Cloud-Services for bug reports and feature requests about the Cloud VPS infrastructure itself

Learn about major near-term plans

Read the News wiki page

Read news and stories about Wikimedia Cloud Services

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)

Revision as of 23:57, 17 February 2020 (view source) imported>JJMC89 m (new key for Category:Toolforge: "Kubernetes" using HotCat) ← Older edit		Revision as of 18:38, 19 February 2020 (view source) imported>SRodlund No edit summary Newer edit →
Line 15:		Line 15:
	Wikiloveslove is a Python 3.7 bot that runs in a Kubernetes deployment. The cronjobs.yaml file that it uses to tell Kubernetes how to start and schedule the bot is reproduced below.		Wikiloveslove is a Python 3.7 bot that runs in a Kubernetes deployment. The cronjobs.yaml file that it uses to tell Kubernetes how to start and schedule the bot is reproduced below.

	After creating the cronjob you can create a test job with <code>kubectl create job --from=cronjob/CRONJOB-NAME test</code> to ~~immediatly~~ trigger the cronjob and then access the logs as usual with <code>kubectl logs job/test -f</code> to debug.		After creating the cronjob you can create a test job with <code>kubectl create job --from=cronjob/CRONJOB-NAME test</code> to immediately trigger the cronjob and then access the logs as usual with <code>kubectl logs job/test -f</code> to debug.

Help:Toolforge/Kubernetes: Difference between revisions

Revision as of 18:38, 19 February 2020

Contents

Overview

Kubernetes webservices

Kubernetes cronjobs

Example cronjob.yaml

Kubernetes continuous jobs

Example deployment.yaml

Monitoring your jobs

Namespaces

Container images

Available container types

PHP

PHP versions & packages

PHP Upgrade

Running Locally

Node.js

Broken npm

Troubleshooting

"failed to create new OS thread" from kubectl

Get a shell inside a running Pod

See also

Navigation menu

Help:Toolforge/Kubernetes: Difference between revisions

Revision as of 18:38, 19 February 2020

Overview

Kubernetes webservices

Kubernetes cronjobs

Example cronjob.yaml

Kubernetes continuous jobs

Example deployment.yaml

Monitoring your jobs

Namespaces

Container images

Available container types

PHP

PHP versions & packages

PHP Upgrade

Running Locally

Node.js

Broken npm

Troubleshooting

"failed to create new OS thread" from kubectl

Get a shell inside a running Pod

See also

Navigation menu

Search