You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
GitLab/Gitlab Runner: Difference between revisions
imported>Jelto (edit monitoring section) |
imported>Jelto No edit summary |
||
Line 1: | Line 1: | ||
GitLab Runner is an application that works with GitLab CI/CD to run jobs in a pipeline.<ref>https://docs.gitlab.com/runner/</ref> For more information see the official [https://docs.gitlab.com/runner/ GitLab Runner documentation]. | GitLab Runner is an application that works with GitLab CI/CD to run jobs in a pipeline.<ref>https://docs.gitlab.com/runner/</ref> For more information see the official [https://docs.gitlab.com/runner/ GitLab Runner documentation]. | ||
=== Current Gitlab Runner setup ([[phab: | === Current Gitlab Runner setup ([[phab:T287279|T287279]]) ===<!-- Additional information needed, feel free to edit --> | ||
We're currently relying on WMCS VPSs for shared runner capacity. There is a project named <code>gitlab-runners</code> in which to provision new instances, and a profile to help provision Docker based runners on those instances. Note that a [[Help:Standalone_puppetmaster|standalone puppetmaster]] in the same project stores the [https://docs.gitlab.com/runner/register/ runner registration token] under <code>/etc/puppet/secret</code>, and Puppet autosigning is turned off to protect the token value. | We're currently relying on WMCS VPSs for shared runner capacity. There is a project named <code>gitlab-runners</code> in which to provision new instances, and a profile to help provision Docker based runners on those instances. Note that a [[Help:Standalone_puppetmaster|standalone puppetmaster]] in the same project stores the [https://docs.gitlab.com/runner/register/ runner registration token] under <code>/etc/puppet/secret</code>, and Puppet autosigning is turned off to protect the token value. | ||
Line 25: | Line 25: | ||
=== Future Gitlab Runner setup ([[phab:T286958|T286958]]) === | === Future Gitlab Runner setup ([[phab:T286958|T286958]]) === | ||
This section contains the requirements and plan for a future Gitlab-Runner setup. The goal is to find a solution which matches our needs for the GitLab Runner infrastructure. GitLab Runner support various | This section contains the requirements and plan for a future Gitlab-Runner setup. The goal is to find a solution which matches our needs for the GitLab Runner infrastructure. GitLab Runner support various platforms, such as Kubernetes, Docker, OpenShift or just Linux VMs. Furthermore a wide range of compute environments can be leveraged, such as WMCS, Ganeti, bare metal hosts or public clouds. So this section compares the different options and collects advantages and disadvantages. Privacy considerations can be found in the next section. | ||
==== GitLab Runner | ==== GitLab Runner platform ==== | ||
GitLab Runner can be installed: | GitLab Runner can be installed: | ||
Line 35: | Line 35: | ||
* in [https://docs.gitlab.com/runner/install/openshift.html OpenShift] | * in [https://docs.gitlab.com/runner/install/openshift.html OpenShift] | ||
* in various other environments | * in various other environments | ||
The follow table compares the most important GitLab Runner | The follow table compares the most important GitLab Runner platforms available (see [https://docs.gitlab.com/runner/install/ Install GitLab Runner]): | ||
{| class="wikitable" | {| class="wikitable" | ||
|+ | |+ | ||
! | !Platform | ||
!Advantages | !Advantages | ||
!Disadvantages | !Disadvantages | ||
Line 49: | Line 49: | ||
| | | | ||
* Low elasticity, difficult to scale | * Low elasticity, difficult to scale | ||
*no separation of jobs | |||
* | * | ||
| | | | ||
Line 84: | Line 85: | ||
|} | |} | ||
==== Compute | ==== Compute Environments ==== | ||
The following table compares the four main computing options for the GitLab Runner setup: WMCS, Ganeti, Bare Metal or Public Cloud. | The following table compares the four main computing options for the GitLab Runner setup: WMCS, Ganeti, Bare Metal or Public Cloud. | ||
{| class="wikitable" | {| class="wikitable" | ||
|+ | |+ | ||
! | !Environment | ||
!Advantages | !Advantages | ||
!Disadvantages | !Disadvantages | ||
Line 135: | Line 136: | ||
===== Elastic demand ===== | ===== Elastic demand ===== | ||
Typically the demand of computing resources for building code and running test is not constant. The usage of CI peaks around African, European and US Timezones and workdays (see Grafana [https://grafana.wikimedia.org/d/000000283/zuul-job?orgId=1&from=now-7d&to=now dashboard] and [https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=contint2001&var-datasource=thanos&var-cluster=ci&from=now-7d&to=now dashboard]). So the ideal solution would adapt to this usage and scale computing resources up and down. This would maximize the utilization of resources and cover usage peaks. However this elasticity comes with costs. In general a dynamic provisioning of Runners is more complex than a static. Currently internal compute | Typically the demand of computing resources for building code and running test is not constant. The usage of CI peaks around African, European and US Timezones and workdays (see Grafana [https://grafana.wikimedia.org/d/000000283/zuul-job?orgId=1&from=now-7d&to=now dashboard] and [https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=contint2001&var-datasource=thanos&var-cluster=ci&from=now-7d&to=now dashboard]). So the ideal solution would adapt to this usage and scale computing resources up and down. This would maximize the utilization of resources and cover usage peaks. However this elasticity comes with costs. In general a dynamic provisioning of Runners is more complex than a static. Currently internal compute environments (such as Ganeti or Bare Metal) have limited elasticity, WMCS is somewhat elastic. So if high elasticity is needed, we have to consider using external providers like GKE. Which opens the discussion about privacy (see next chapter) and being independent from external parties. | ||
So even if a elastic Runner setup would be the better technical solution we have to ask if we really need high elasticity now. | So even if a elastic Runner setup would be the better technical solution we have to ask if we really need high elasticity now. | ||
Line 171: | Line 172: | ||
* Is a GitLab Runner Kubernetes cluster the best way to accumulate Kubernetes knowledge? | * Is a GitLab Runner Kubernetes cluster the best way to accumulate Kubernetes knowledge? | ||
<references /> | <references /> | ||
=== Requirements for new CI/CD system === | |||
Currently the Foundation is running multiple tools for CI/CD purpose. Some of them are [[mw:Continuous_integration/Jenkins|Jenkins]], [[mw:Continuous_integration/Zuul|Zuul]] and [[mw:Gerrit|Gerrit]]. So when choosing a environment and a platform for the new GitLab Runner setup we should also take in to consideration what CI/CD capabilities are used. We should also think about what capabilities we may use in the future. | |||
* Jenkins is mostly running testing, linting and diffing tasks(?) | |||
* Jenkins also runs some build tasks(?) | |||
** Docker Images(?) | |||
** Debian packages(?) | |||
** Compiling code(?) | |||
* Gerrit is used for version control and code reviewing | |||
* Zuul is triggering jobs in Jenkins when changes are detected in Gerrit(?) |
Revision as of 13:00, 23 September 2021
GitLab Runner is an application that works with GitLab CI/CD to run jobs in a pipeline.[1] For more information see the official GitLab Runner documentation.
Current Gitlab Runner setup (T287279)
We're currently relying on WMCS VPSs for shared runner capacity. There is a project named gitlab-runners
in which to provision new instances, and a profile to help provision Docker based runners on those instances. Note that a standalone puppetmaster in the same project stores the runner registration token under /etc/puppet/secret
, and Puppet autosigning is turned off to protect the token value.
To set up a new shared runner, following these steps.
- Create a new WMCS VPS instance.
- Log in to [1] and navigate to the
gitlab-runners
project. - Launch a new Debian
buster
instance, following therunner-{nnnn}
naming convention. - Add
profile::gitlab::runner
to the instance's Puppet Classes under the Puppet Configuration tab.
- Log in to [1] and navigate to the
- Wait until the new instance has fully provisioned and you can successfully
ssh
to the running instance using your authorized key. (This typically takes a few minutes.) - Do the little SSL dance that is required of instances that use a standalone puppetmaster.
- On the new runner (
runner-{nnnn}.gitlab-runners.eqiad1.wikimedia.cloud
).- Run
sudo rm -rf /var/lib/puppet/ssl
to remove the existing SSL certs used by the default puppetmaster. - Run
sudo -i puppet agent --test --verbose
to have the puppet client generate a new SSL cert.
- Run
- On
gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud
sign the new instance's SSL cert.- Run
sudo -i puppet cert list
and find the new instance in the list. - Run
sudo -i puppet cert sign runner-{nnnn}.gitlab-runners.eqiad1.wikimedia.cloud
to sign the client cert.
- Run
- On the new runner (
- Run
sudo -i puppet agent --test --verbose
on the runner to ensure it has fully provisioned theprofile::gitlab::runner
profile. - Verify that the runner has successfully registered with our GitLab instance by viewing the runner list.
Future Gitlab Runner setup (T286958)
This section contains the requirements and plan for a future Gitlab-Runner setup. The goal is to find a solution which matches our needs for the GitLab Runner infrastructure. GitLab Runner support various platforms, such as Kubernetes, Docker, OpenShift or just Linux VMs. Furthermore a wide range of compute environments can be leveraged, such as WMCS, Ganeti, bare metal hosts or public clouds. So this section compares the different options and collects advantages and disadvantages. Privacy considerations can be found in the next section.
GitLab Runner platform
GitLab Runner can be installed:
- in Linux (using official packages)
- in container
- in Kubernetes (as helm chart or agent)
- in OpenShift
- in various other environments
The follow table compares the most important GitLab Runner platforms available (see Install GitLab Runner):
Platform | Advantages | Disadvantages | Additional considerations |
---|---|---|---|
Linux |
|
|
|
Container |
|
|
|
Kubernetes |
|
|
|
OpenShift |
|
Compute Environments
The following table compares the four main computing options for the GitLab Runner setup: WMCS, Ganeti, Bare Metal or Public Cloud.
Environment | Advantages | Disadvantages | Additional considerations |
---|---|---|---|
WMCS |
|
|
|
Ganeti |
|
|
|
Bare metal |
|
|
|
Public Cloud (e.g. GCP) |
|
|
|
Elastic demand
Typically the demand of computing resources for building code and running test is not constant. The usage of CI peaks around African, European and US Timezones and workdays (see Grafana dashboard and dashboard). So the ideal solution would adapt to this usage and scale computing resources up and down. This would maximize the utilization of resources and cover usage peaks. However this elasticity comes with costs. In general a dynamic provisioning of Runners is more complex than a static. Currently internal compute environments (such as Ganeti or Bare Metal) have limited elasticity, WMCS is somewhat elastic. So if high elasticity is needed, we have to consider using external providers like GKE. Which opens the discussion about privacy (see next chapter) and being independent from external parties.
So even if a elastic Runner setup would be the better technical solution we have to ask if we really need high elasticity now.
Further reading:
https://docs.gitlab.com/runner/executors/kubernetes.html
https://docs.gitlab.com/runner/configuration/autoscale.html
Privacy considerations
Privacy is one core principal of WMF. So if public clouds are used we have to make sure this usage aligns with our privacy policy and doesn't cause any security risks.
We have to think about what data is transmitted to public clouds during builds and tests. Do we include secrets, passwords or private user data when running a job? Do we need a special policy for CI variables and secrets? Do we consider this data leaked/compromised when transmitted to public cloud machines even when encrypted/restricted machines are used?
We also have to think about how to secure the artifacts and test results of jobs running in public clouds. How do we implement trust? How do we check if artifacts (images, compiled code) or test results weren't compromised?
Monitoring of performance and usage
Gitlab-Runner support Prometheus metric export. This metrics and some Grafana dashboards should give insights in performance and usage. See Monitoring Gitlab Runner documentation.
However the Gitlab Runner exporter does not support authorization or https. So depending on where the Runners are hosted, a https proxy with authorization is required.
Open Questions:
Open questions:
- Do we need high elasticity now/in the near future?
- Do we have any drawbacks with the current, non-elastic setup with static Jenkins machines?
- Do we want to open a discussion about public cloud usage and privacy?
- Do we want to be depended on public cloud offerings?
- Do we have the resources to plan, implement and maintain a additional Kubernetes Cluster?
- Is a GitLab Runner Kubernetes cluster the best way to accumulate Kubernetes knowledge?
Requirements for new CI/CD system
Currently the Foundation is running multiple tools for CI/CD purpose. Some of them are Jenkins, Zuul and Gerrit. So when choosing a environment and a platform for the new GitLab Runner setup we should also take in to consideration what CI/CD capabilities are used. We should also think about what capabilities we may use in the future.
- Jenkins is mostly running testing, linting and diffing tasks(?)
- Jenkins also runs some build tasks(?)
- Docker Images(?)
- Debian packages(?)
- Compiling code(?)
- Gerrit is used for version control and code reviewing
- Zuul is triggering jobs in Jenkins when changes are detected in Gerrit(?)