Difference between revisions of "GitLab/Gitlab Runner"

From Wikitech-static
Jump to navigation Jump to search
imported>Jelto
 
imported>Dduvall
(→‎Setting up a new shared runner: Verified that running puppet agent twice at the end is not strictly necessary.)
Line 1: Line 1:
GitLab Runner is an application that works with GitLab CI/CD to run jobs in a pipeline.<ref>https://docs.gitlab.com/runner/</ref> For more information see the official [https://docs.gitlab.com/runner/ GitLab Runner documentation].
GitLab Runner is an application that works with GitLab CI/CD to run jobs in a pipeline.<ref>https://docs.gitlab.com/runner/</ref> For more information see the official [https://docs.gitlab.com/runner/ GitLab Runner documentation].


=== Current Gitlab Runner setup ([[phab:T287279|T287279]]) ===<!-- Additional information needed, feel free to edit -->
=== Current Gitlab Runner setup ([[phab:T287504|T287504]]) ===<!-- Additional information needed, feel free to edit -->


In WMCS there is a project <code>gitlab-runners</code> to host a set of shared GitLab runners. Currently the runners are installed and configured with [[gerrit:plugins/gitiles/operations/gitlab-ansible/|Ansible playbook]]. To add runners, add the hosts to <code>[gitlab_runners]</code> section of 'hosts' file. Then run the Ansible Playbook for gitlab-runners. The command is wrapped in <code>[[gerrit:plugins/gitiles/operations/gitlab-ansible/+/refs/heads/master/install-gitlab-runner.sh|install-gitlab-runner.sh]]</code> script.<ref>https://phabricator.wikimedia.org/T285913
We're currently relying on WMCS VPSs for shared runner capacity. There is a project named <code>gitlab-runners</code> in which to provision new instances, and a profile to help provision Docker based runners on those instances. Note that a [[Help:Standalone_puppetmaster|standalone puppetmaster]] in the same project stores the [https://docs.gitlab.com/runner/register/ runner registration token] under <code>/etc/puppet/secret</code>, and Puppet autosigning is turned off to protect the token value.


https://phabricator.wikimedia.org/T287279</ref>
==== Setting up a new shared runner ====


There are efforts to move all of the installation and configuration to puppet: [[phab:T287279#7237735|T287279#7237735]].
To set up a new shared runner, following these steps.
 
# Create a new WMCS VPS instance.
## Log in to [https://horizon.wikimedia.org] and navigate to the <code>gitlab-runners</code> project.
## Launch a new Debian <code>buster</code> instance, following the <code>runner-{nnnn}</code> naming convention.
## Add <code>profile::gitlab::runner</code> to the instance's Puppet Classes under the Puppet Configuration tab.
# Wait until the new instance has fully provisioned and you can successfully <code>ssh</code> to the running instance using your authorized key. (This typically takes a few minutes.)
# Do [[Help:Standalone_puppetmaster#Step_2:_Setup_a_puppet_client|the little SSL dance]] that is required of instances that use a standalone puppetmaster.
## On the new runner (<code>runner-{nnnn}.gitlab-runners.eqiad1.wikimedia.cloud</code>).
### Run <code>sudo rm -rf /var/lib/puppet/ssl</code> to remove the existing SSL certs used by the default puppetmaster.
### Run <code>sudo -i puppet agent --test --verbose</code> to have the puppet client generate a new SSL cert.
## On <code>gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud</code> sign the new instance's SSL cert.
### Run <code>sudo -i puppet cert list</code> and find the new instance in the list.
### Run <code>sudo -i puppet cert sign runner-{nnnn}.gitlab-runners.eqiad1.wikimedia.cloud</code> to sign the client cert.
# Run <code>sudo -i puppet agent --test --verbose</code> on the runner to ensure it has fully provisioned the <code>profile::gitlab::runner</code> profile.
# Verify that the runner has successfully registered with our GitLab instance by viewing the [https://gitlab.wikimedia.org/admin/runners runner list].


=== Future Gitlab Runner setup ([[phab:T286958|T286958]]) ===
=== Future Gitlab Runner setup ([[phab:T286958|T286958]]) ===
This section contains the requirements and plan for a future Gitlab-Runner setup. The goal is to find a secure, scalable and easy to build and maintain setup for the GitLab Runner infrastructure.
This section contains the requirements and plan for a future Gitlab-Runner setup. The goal is to find a secure, scalable and easy to build and maintain setup for the GitLab Runner infrastructure. GitLab Runner support various environments, such as Kubernetes, Docker, OpenShift or just Linux VMs. Furthermore a wide range of compute platforms can be leveraged, such as WMCS, Ganetti, bare metal hosts or public clouds. So this section tries to compare the different options and collect advantages and disadvantages. Privacy considerations can be found in the next section.


==== Compute Resources ====
==== GitLab Runner environments ====
GitLab Runner can run on various platforms, such as Kubernetes, Docker, OpenShift or just Linux VMs. Furthermore a wide range of compute platforms can be leveraged, such as WMCS, Ganetti, bare metal hosts or public clouds. So this section tries to compare the different options and collect advantages and disadvantages. Privacy considerations can be found in the next section.
GitLab Runner can be installed:
 
* in Linux (using [https://docs.gitlab.com/runner/install/linux-repository.html official packages])
* in [https://docs.gitlab.com/runner/install/docker.html container]
* in Kubernetes (as [https://docs.gitlab.com/runner/install/kubernetes.html helm chart] or [https://docs.gitlab.com/runner/install/kubernetes-agent.html agent])
* in [https://docs.gitlab.com/runner/install/openshift.html OpenShift]
* in various other environments
The follow table tries to compare the most important GitLab Runner environments available:
{| class="wikitable"
|+
!Environment
!Advantages
!Disadvantages
!Additional considerations
|-
|Linux
|
* Easy to setup
* low maintenance
* Similar to current solution
|
* Low elasticity, difficult to scale
*
|
|-
|Container
|
* Easy to setup
* low maintenance
|
* difficult to scale
* auto scaling needs<code>docker-machine</code>
|
|-
|Kubernetes
|
* High elasticity/auto scaling
*
|
* Additional Kubernetes needed (for security)
* Additional cluster needs maintenance
* More difficult to setup
|
* Could be used to strengthen Kubernetes knowledge
* Auto scaling needs elastic compute plattform
*Maybe a general purpose non-production cluster can be build?
|-
|OpenShift
|
|
* not in use in WMF
|
|}
 
==== Compute Plattforms ====
{| class="wikitable"
|+
!Plattform
!Advantages
!Disadvantages
!Additional considerations
|-
|WMCS
|
|
|
|-
|Ganetti
|
|
|
|-
|Bare metal
|
|
|
|-
|Public Cloud (e.g. GCP)
|
|
|
|}


===== Elastic demand =====
===== Elastic demand =====
https://docs.gitlab.com/runner/enterprise_guide/#autoscaling-configuration-one-or-more-runner-managers-multiple-workers
https://docs.gitlab.com/runner/executors/kubernetes.html
https://docs.gitlab.com/runner/configuration/autoscale.html
https://docs.gitlab.com/runner/configuration/autoscale.html



Revision as of 18:09, 2 September 2021

GitLab Runner is an application that works with GitLab CI/CD to run jobs in a pipeline.[1] For more information see the official GitLab Runner documentation.

Current Gitlab Runner setup (T287504)

We're currently relying on WMCS VPSs for shared runner capacity. There is a project named gitlab-runners in which to provision new instances, and a profile to help provision Docker based runners on those instances. Note that a standalone puppetmaster in the same project stores the runner registration token under /etc/puppet/secret, and Puppet autosigning is turned off to protect the token value.

Setting up a new shared runner

To set up a new shared runner, following these steps.

  1. Create a new WMCS VPS instance.
    1. Log in to [1] and navigate to the gitlab-runners project.
    2. Launch a new Debian buster instance, following the runner-{nnnn} naming convention.
    3. Add profile::gitlab::runner to the instance's Puppet Classes under the Puppet Configuration tab.
  2. Wait until the new instance has fully provisioned and you can successfully ssh to the running instance using your authorized key. (This typically takes a few minutes.)
  3. Do the little SSL dance that is required of instances that use a standalone puppetmaster.
    1. On the new runner (runner-{nnnn}.gitlab-runners.eqiad1.wikimedia.cloud).
      1. Run sudo rm -rf /var/lib/puppet/ssl to remove the existing SSL certs used by the default puppetmaster.
      2. Run sudo -i puppet agent --test --verbose to have the puppet client generate a new SSL cert.
    2. On gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud sign the new instance's SSL cert.
      1. Run sudo -i puppet cert list and find the new instance in the list.
      2. Run sudo -i puppet cert sign runner-{nnnn}.gitlab-runners.eqiad1.wikimedia.cloud to sign the client cert.
  4. Run sudo -i puppet agent --test --verbose on the runner to ensure it has fully provisioned the profile::gitlab::runner profile.
  5. Verify that the runner has successfully registered with our GitLab instance by viewing the runner list.

Future Gitlab Runner setup (T286958)

This section contains the requirements and plan for a future Gitlab-Runner setup. The goal is to find a secure, scalable and easy to build and maintain setup for the GitLab Runner infrastructure. GitLab Runner support various environments, such as Kubernetes, Docker, OpenShift or just Linux VMs. Furthermore a wide range of compute platforms can be leveraged, such as WMCS, Ganetti, bare metal hosts or public clouds. So this section tries to compare the different options and collect advantages and disadvantages. Privacy considerations can be found in the next section.

GitLab Runner environments

GitLab Runner can be installed:

The follow table tries to compare the most important GitLab Runner environments available:

Environment Advantages Disadvantages Additional considerations
Linux
  • Easy to setup
  • low maintenance
  • Similar to current solution
  • Low elasticity, difficult to scale
Container
  • Easy to setup
  • low maintenance
  • difficult to scale
  • auto scaling needsdocker-machine
Kubernetes
  • High elasticity/auto scaling
  • Additional Kubernetes needed (for security)
  • Additional cluster needs maintenance
  • More difficult to setup
  • Could be used to strengthen Kubernetes knowledge
  • Auto scaling needs elastic compute plattform
  • Maybe a general purpose non-production cluster can be build?
OpenShift
  • not in use in WMF

Compute Plattforms

Plattform Advantages Disadvantages Additional considerations
WMCS
Ganetti
Bare metal
Public Cloud (e.g. GCP)
Elastic demand

https://docs.gitlab.com/runner/enterprise_guide/#autoscaling-configuration-one-or-more-runner-managers-multiple-workers

https://docs.gitlab.com/runner/executors/kubernetes.html

https://docs.gitlab.com/runner/configuration/autoscale.html

Privacy considerations

Whether these can safely run on a third-party platform

Monitoring of performance and usage

Gitlab-Runner export Prometheus metrics. This metrics should give insights in performance and usage. See Monitoring Gitlab Runner documentation.

However the Gitlab Runner exporter does not support authorization or https. So depending on where the Runners are hosted, a https proxy with authorization is required.

[2]