You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Portal:Toolforge/Admin/Kubernetes/Components
This page contains information on how to operate, build and deploy various Toolforge Kubernetes components.
The information in here is subject to change as we move in the direction of Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_Kubernetes_component_workflow_improvements
In general, assume this is valid for both tools and toolsbeta.
Workflow
The workflow as we have it today.
Steps
This is what you normally would do:
code
: make changes to a given code repository, push & merge as usual.build
: once code is ready to deploy to k8s, if the component needs it, build the container image using the cookbook.pre-deploy
: if the component uses a built container image, chances are that the image tag requires a refresh (avoid :latest). This is a new, separate commit, pointing to the newest built container image from previous step.deploy
: once ready to deploy to k8s for real, use the cookbook.verify
test by hand, verify, etc.rollback
if a rollback is needed, follow again the same steps but merging a revert commit plus (if required) an image tag bump.
Build
Use the wmcs.toolforge.k8s.component.build
cookbook to build the docker container image that will later be deployed in the kubernetes cluster.
user@laptop:~$ cookbook wmcs.toolforge.k8s.component.build -h
usage: cookbooks.wmcs.toolforge.k8s.component.build [-h] [--project PROJECT] [--task-id TASK_ID] [--no-dologmsg]
[--registry-url REGISTRY_URL]
[--docker-builder-hostname DOCKER_BUILDER_HOSTNAME] --git-url GIT_URL
[--git-name GIT_NAME] [--git-branch GIT_BRANCH]
[--docker-image-tag DOCKER_IMAGE_TAG]
[--docker-image-name DOCKER_IMAGE_NAME]
WMCS Toolforge Kubernetes - build a docker image for a custom component
Usage example:
cookbook wmcs.toolforge.k8s.component/build \
--git-url https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api
options:
-h, --help show this help message and exit
--project PROJECT Relevant Cloud VPS openstack project (for operations, dologmsg, etc). If this cookbook is for hardware,
this only affects dologmsg calls. Default is 'tools'.
--task-id TASK_ID Id of the task related to this operation (ex. T123456). (default: None)
--no-dologmsg To disable dologmsg calls (no SAL messages on IRC). (default: False)
--registry-url REGISTRY_URL
docker registry URL (default: docker-registry.tools.wmflabs.org)
--docker-builder-hostname DOCKER_BUILDER_HOSTNAME
docker image builder virtual machine hostname (default: tools-docker-imagebuilder-01)
--git-url GIT_URL git URL for the source code (default: None)
--git-name GIT_NAME git repository name. If not provided, it will be guessed based on the git URL (default: None)
--git-branch GIT_BRANCH
git branch in the source repository (default: main)
--docker-image-tag DOCKER_IMAGE_TAG
docker tag for the new image, if not provided the git hash of the latest commit will be used (default:
None)
--docker-image-name DOCKER_IMAGE_NAME
docker image name. If not provided, it will be guessed based on the git name (default: None)
Example:
user@laptop:~$ cookbook wmcs.toolforge.k8s.component.build --git-url https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api
[..]
user@laptop:~$ cookbook wmcs.toolforge.k8s.component.build --git-url https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer
INFO: guesses git tree name as jobs-framework-emailer
INFO: guesses docker image name as toolforge-jobs-framework-emailer
START - Cookbook wmcs.toolforge.k8s.component.build
INFO: using build node tools-docker-imagebuilder-01.tools.eqiad1.wikimedia.cloud
INFO: creating temp dir /tmp/cookbook-toolforge-k8s-component-build-ulprobicib
INFO: git cloning https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer
INFO: git checkout main on cloning /tmp/cookbook-toolforge-k8s-component-build-ulprobicib/jobs-framework-emailer
INFO: building docker image toolforge-jobs-framework-emailer:latest
INFO: cleaning up temp dir /tmp/cookbook-toolforge-k8s-component-build-ulprobicib
INFO: creating docker tag docker-registry.tools.wmflabs.org/toolforge-jobs-framework-emailer:latest
INFO: pushing to the registry docker-registry.tools.wmflabs.org/toolforge-jobs-framework-emailer:latest
[DOLOGMSG]: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-emailer:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer (084ee51)
END (PASS) - Cookbook wmcs.toolforge.k8s.component.build (exit_code=0)
Deploy
Use the wmcs.toolforge.k8s.component.deploy
cookbook to deploy the custom component in the given kubernetes cluster.
user@laptop:~$ cookbook wmcs.toolforge.k8s.component.deploy -h
usage: cookbooks.wmcs.toolforge.k8s.component.deploy [-h] --cluster-name {tools,toolsbeta} [--task-id TASK_ID] [--no-dologmsg] --git-url GIT_URL
[--git-name GIT_NAME] [--git-branch GIT_BRANCH] [--deployment-command DEPLOYMENT_COMMAND]
WMCS Toolforge Kubernetes - deploy a kubernetes custom component
Usage example:
cookbook wmcs.toolforge.k8s.component.deploy \
--git-url https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api
options:
-h, --help show this help message and exit
--cluster-name {tools,toolsbeta}
cluster to work on (default: None)
--task-id TASK_ID Id of the task related to this operation (ex. T123456). (default: None)
--no-dologmsg To disable dologmsg calls (no SAL messages on IRC). (default: False)
--git-url GIT_URL git URL for the source code (default: None)
--git-name GIT_NAME git repository name. If not provided, it will be guessed based on the git URL (default: None)
--git-branch GIT_BRANCH
git branch in the source repository (default: main)
--deployment-command DEPLOYMENT_COMMAND
command to trigger the deployment. (default: ./deploy.sh)
List
This is a list of our custom components and where to find them.
This is the only known source of truth for this information. Even if the list might drift outdated, as of this writing there is no other place in which this information is collected.
Third-party components
Additional consideration for some third-party components that we use.
Name | Deployment repository | Current version | K8s versions supported |
---|---|---|---|
Calico | https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/ | 3.24.5 | 1.22 - 1.25[1] |
ingress-nginx | https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx | 1.4.0 (chart-4.3.0) | 1.22 - 1.25[2] |
kube-state-metrics | https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics | 2.2.4 (chart-4.1.0) | ? |
metrics-server | https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics | 0.6.2 (chart-3.8.3) | 1.19+[3] |
cadvisor | https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics | 0.30.2 | n/a (not a Kubernetes specific tool) |
Prometheus | https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics | n/a | |
cert-manager | https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager | 1.11.0 | 1.21 - 1.26[4] |
reloader | https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager | 1.0.5 | ? |
Tekton Pipelines | https://github.com/toolforge/buildservice/tree/main/deploy/base-tekton | 0.33.2 | 1.20+[5] |
References
- ↑ https://docs.tigera.io/archive/v3.24/getting-started/kubernetes/requirements
- ↑ https://github.com/kubernetes/ingress-nginx/tree/main#supported-versions-table
- ↑ https://github.com/kubernetes-sigs/metrics-server#compatibility-matrix
- ↑ https://cert-manager.io/docs/installation/supported-releases/
- ↑ https://tekton.dev/vault/pipelines-v0.33.x/install/
See also
- Wikimedia_Cloud_Services_team/EnhancementProposals/Decision_record_T303931_k8s_standard_deployment_code_pattern
- Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_Kubernetes_component_workflow_improvements
- Portal:Toolforge/Admin/Kubernetes/labels
- Phabricator T291915 - toolforge: automate how we deploy custom k8s components
- Portal:Toolforge/Admin/Kubernetes#Docker_Images
- Portal:Toolforge/Admin/Kubernetes/Docker-registry
- Portal:Toolforge/Admin/lima-kilo