GitLab/Gitlab Runner
| GitLab Runner |
|---|
|
|
GitLab Runner is an application that works with GitLab to run CI/CD jobs in a pipeline. [ 1 ] For more information see the official GitLab Runner documentation . For information on how GitLab runners are used in production deployments, see Deploying services to production .
GitLab Runner types
Our GitLab instance offers different types of CI Runners:
- Cloud Runners
-
The default instance-wide CI workers. They are running on a managed Kubernetes in Digital Ocean. They are available instance wide for every project. Cloud Runners use the tag
cloud(andkubernetes) but accept untagged jobs as well. Jobs running on these runners have 2 vcpu and 1GiB of memory available to them. - Memory-optimized Cloud Runners
-
These behave just like regular Cloud runners but feature more RAM than their counterpart. These runners are useful for more intensive
building/packaging jobs
. To utilize these runners, add the
memory-optimizedtag. Jobs running on these runners have 2.1 vcpu and 5.7GiB of memory available to them. - Shared GitLab Runners
-
General purpose CI workers. These Runners execute jobs for a wide range of projects inside the
/reposgroup in GitLab. If access to this kind of Runner is needed, consider moving your git repository to the/reposgroup and make yourself familiar with the details under Shared GitLab Runners . Shared Runners use the tagwmcsbut also accept untagged jobs. These runners have 8 vcpu and 24 GiB of memory available to them (shared with other jobs that may be running on the node at the same time). - Trusted GitLab Runners
-
These offer a platform for CI jobs with additional security needs (like building production artifacts). These Runners run in WMF production infrastructure and access to these Runners is gated and restricted. Access has to be requested on per-project basis, so please take a look on
Trusted GitLab Runners
on how to get access. Trusted Runners use the tag
trustedand do not accept untagged jobs.
Permission matrix
The following table explains which features and permissions are available for which type of runner.
| Runner Type | Trusted Runners | Dockerfile Runners | Shared Runner | Cloud Runners |
|---|---|---|---|---|
| access level | access has to be requested per project in projects.json | access has to be requested per project in projects.json and is restricted to a small set of infrastructure/foundational container projects | all projects in /repos have access | all projects have access |
| allowed images |
only images from WMF registry
docker-registry.wikimedia.org/**/*
|
only images from WMF registry
docker-registry.wikimedia.org/**/*
|
images from WMF registry, popular upstream Dockerhub images (like debian, golang, node, python), GitLab upstream images, wmflabs images. See
profile::gitlab::runner::allowed_images
in
operations/puppet/hieradata/cloud.yaml
|
no restriction |
| allowed Container build frontends | blubber frontend | blubber and Dockerfile frontend | blubber frontend | blubber and Dockerfile frontend |
| publish images to wmf registry | Yes (using credentials from jwt-authorizer ) | Yes (using credentials from jwt-authorizer ) | No | No |
| allowed job sources | Jobs from protected tags and branches only | Jobs from protected tags and branches only | Jobs from all branches or tags | Jobs from all branches or tags |
| allowed WMF services |
GitLab, apt host, doc host, docker registry, Kubernetes staging (see
profile::gitlab::runner::allowed_services
)
|
GitLab, apt host, doc host, docker registry, Kubernetes staging (see
profile::gitlab::runner::allowed_services
)
|
GitLab, Zuul | none |
| allowed external services | Https services using the webproxy , Dockerhub is blocked | Https services using the webproxy | All public https services | All public https services |
Adding new runners (end-users)
For special use-cases new runners can be added to any project by end-users. The token needed to register the runner can be obtained under project Settings > CI/CD > Runners (expand). Use the button "New project runner" to add a runner. The registration token will be shown once, so store the token in a secure place. Usage of the registration token varies depending on the type of runner (Docker, VM, Kubernetes). See register a runner upstream documentation. Please note this runners are self-managed, so you have to make sure to install updates and stay compatible with the GitLab server version.
Adding new runners (administrators)
The process of adding Shared, Trusted or instance-wide runners is different from adding self-managed project runners. The tokens and registration are created programmatically to ensure all runners use the correct configuration (protection status, tags, description and untagged jobs). The tool used to create the tokens is https://gitlab.wikimedia.org/repos/releng/gitlab-settings . The actual registration happens in Puppet or Terraform, depending on the Runner.
To generate new tokens, use the
runners generate-tokens
command. The script will walk you over every runner configured in
runner-config.json
. Please note the output of the script contains private tokens.
Example run:
./runners generate-tokens
Please add the following tokens to private puppet.
The token should be added as profile::gitlab::runner::token in either private puppet or WMCS puppet master.
The token for cloud runners should be stored as TF_VAR_runner_token in https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/settings/ci_cd.
The tokens are secrets, do not store them anywhere else!
Do you want to create a new token for shared_runners [y/N]: n
Skipping shared_runners
Do you want to create a new token for trusted_runners [y/N]: y
trusted_runners token: <removed>
As mentioned in the script output, the tokens are then used in Puppet or GitLab CI deployments. So for Shared and Trusted Runners configured by puppet, the token has to added as
profile::gitlab::runner::token
to
private puppet
. Runners in Digital Ocean use Terraform for deployments, so the token is added as
TF_VAR_runner_token
in
https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/settings/ci_cd
.
New Runners should pick up a updated token with the next puppet run or Terraform deployment. Existing Runners have to be un-registered and re-registered manually by setting
profile::gitlab::runner::ensure
to
absent
, run puppet and setting it to
present
again. Also the script does not rotate old tokens, it just creates to tokens. So manual deletion of Runners in
https://gitlab.wikimedia.org/admin/runners
may be required.
Evaluation and Design
Evaluation sub-pages on the right menu offer more insights into the design and security considerations.