imported>Jelto |
imported>Jelto |
| (7 intermediate revisions by 2 users not shown) |
| Line 1: |
Line 1: |
| GitLab Runner is an application that works with GitLab CI/CD to run jobs in a pipeline.<ref>https://docs.gitlab.com/runner/</ref> For more information see the official [https://docs.gitlab.com/runner/ GitLab Runner documentation]. | | {{Sidebar |
| | | style = background: white; padding:10px; padding-{{dir|{{pagelang}}|left|right}}:13px; margin:{{dir|{{pagelang}}|5px 12px 5px 0|5px 0 5px 12px}}; width: 350px; |
| | | name = GitLab Runner |
| | | title = GitLab Runner |
| | | image = [[File:Gitlab-logo.svg.svg|center|250px]] |
| | | headingstyle = font-size: 130%; padding: .5em; |
| | | contentstyle = text-align: {{dir|{{pagelang}}|right|left}}; font-size: 14px; padding: .5em; line-height: 1.5; |
| | | abovestyle = text-align: {{dir|{{pagelang}}|right|left}}; |
| | | content1 = |
| | {{Special:PrefixIndex/{{FULLPAGENAME}}/ |hideredirects=1 |stripprefix=1}} |
| | * External resources: |
| | ** [https://gitlab.wikimedia.org/admin/runners GitLab Runner Admin menu] |
| | ** [https://grafana.wikimedia.org/d/Chb-gC07k/gitlab-ci-overview?orgId=1 GitLab CI metrics] |
| | }} |
|
| |
|
| === Current Gitlab Runner setup ([[phab:T287279|T287279]]) ===<!-- Additional information needed, feel free to edit -->
| | GitLab Runner is an application that works with GitLab CI/CD to run jobs in a pipeline.<ref>https://docs.gitlab.com/runner/</ref> For more information see the official [https://docs.gitlab.com/runner/ GitLab Runner documentation]. |
|
| |
|
| We're currently relying on WMCS VPSs for shared runner capacity. There is a project named <code>gitlab-runners</code> in which to provision new instances, and a profile to help provision Docker based runners on those instances. Note that a [[Help:Standalone_puppetmaster|standalone puppetmaster]] in the same project stores the [https://docs.gitlab.com/runner/register/ runner registration token] under <code>/etc/puppet/secret</code>, and Puppet autosigning is turned off to protect the token value.
| | ===== GitLab Runner types ===== |
| | GitLab offers different types of CI Runners. [[GitLab/Gitlab Runner/Shared Runners|Shared GitLab Runners]] are general purpos CI workers. This Runners execute jobs for a wide range of projects inside the <code>[https://gitlab.wikimedia.org/repos /repos]</code> group in GitLab. If access to this kind of Runners is needed, consider moving to the <code>[https://gitlab.wikimedia.org/repos /repos]</code> group and make yourself familiar with the details under [[GitLab/Gitlab Runner/Shared Runners|Shared GitLab Runners]]. |
|
| |
|
| ==== Setting up a new shared runner ====
| | [[GitLab/Gitlab Runner/Trusted Runners|Trusted GitLab Runners]] offer a platform for CI jobs with additional security needs (like building production artifacts). This Runners live inside WMF infrastructure and access to this Runners is gated and restricted. Access has to be requested on project basis, so please take a look on [[GitLab/Gitlab Runner/Trusted Runners|Trusted GitLab Runners]] on how to get access. |
|
| |
|
| To set up a new shared runner, following these steps.
| | It is planned to add CI support for all projects using [[GitLab/Gitlab Runner/Cloud Runners|Cloud Runners]]. This Runners are in design phase and access to this Runners will be announced. |
|
| |
|
| # Create a new WMCS VPS instance.
| | ===== Evaluation and Design ===== |
| ## Log in to [https://horizon.wikimedia.org] and navigate to the <code>gitlab-runners</code> project.
| | Evaluation sub-pages on the right menu offer more insights into the design and security considerations. |
| ## Launch a new Debian <code>buster</code> instance, following the <code>runner-{nnnn}</code> naming convention.
| |
| ## Add <code>profile::gitlab::runner</code> to the instance's Puppet Classes under the Puppet Configuration tab.
| |
| # Wait until the new instance has fully provisioned and you can successfully <code>ssh</code> to the running instance using your authorized key. (This typically takes a few minutes.)
| |
| # Do [[Help:Standalone_puppetmaster#Step_2:_Setup_a_puppet_client|the little SSL dance]] that is required of instances that use a standalone puppetmaster.
| |
| ## On the new runner (<code>runner-{nnnn}.gitlab-runners.eqiad1.wikimedia.cloud</code>).
| |
| ### Run <code>sudo rm -rf /var/lib/puppet/ssl</code> to remove the existing SSL certs used by the default puppetmaster.
| |
| ### Run <code>sudo -i puppet agent --test --verbose</code> to have the puppet client generate a new SSL cert.
| |
| ## On <code>gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud</code> sign the new instance's SSL cert.
| |
| ### Run <code>sudo -i puppet cert list</code> and find the new instance in the list.
| |
| ### Run <code>sudo -i puppet cert sign runner-{nnnn}.gitlab-runners.eqiad1.wikimedia.cloud</code> to sign the client cert.
| |
| # Run <code>sudo -i puppet agent --test --verbose</code> on the runner to ensure it has fully provisioned the <code>profile::gitlab::runner</code> [[gerrit:plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/gitlab/runner.pp|profile]].
| |
| # Verify that the runner has successfully registered with our GitLab instance by viewing the [https://gitlab.wikimedia.org/admin/runners runner list].
| |
|
| |
|
| === Future Gitlab Runner setup ([[phab:T286958|T286958]]) ===
| |
| This section contains the requirements and plan for a future Gitlab-Runner setup. The goal is to find a solution which matches our needs for the GitLab Runner infrastructure. GitLab Runner support various platforms, such as Kubernetes, Docker, OpenShift or just Linux VMs. Furthermore a wide range of compute environments can be leveraged, such as WMCS, Ganeti, bare metal hosts or public clouds. So this section compares the different options and collects advantages and disadvantages. Privacy considerations can be found in the next section.
| |
| [[File:Future Gitlab Runner Setup.png|thumb|427x427px]]
| |
|
| |
| ==== GitLab Runner platform ====
| |
| GitLab Runner can be installed:
| |
|
| |
| * in Linux (using [https://docs.gitlab.com/runner/install/linux-repository.html official packages])
| |
| * in [https://docs.gitlab.com/runner/install/docker.html container]
| |
| * in Kubernetes (as [https://docs.gitlab.com/runner/install/kubernetes.html helm chart] or [https://docs.gitlab.com/runner/install/kubernetes-agent.html agent])
| |
| * in [https://docs.gitlab.com/runner/install/openshift.html OpenShift]
| |
| * in various other environments
| |
| The follow table compares the most important GitLab Runner platforms available (see [https://docs.gitlab.com/runner/install/ Install GitLab Runner]):
| |
| {| class="wikitable"
| |
| |+
| |
| !Platform
| |
| !Advantages
| |
| !Disadvantages
| |
| !Additional considerations
| |
| |-
| |
| |Linux
| |
| |
| |
| * Easy to setup
| |
| * low maintenance
| |
| |
| |
| * Low elasticity, difficult to scale
| |
| *no separation of jobs<ref>https://docs.gitlab.com/runner/security/</ref>
| |
| *
| |
| |
| |
| |-
| |
| |Container
| |
| |
| |
| * Easy to setup
| |
| * low maintenance
| |
| *separation of jobs by containers
| |
| *Similar to current solution
| |
| |
| |
| * difficult to scale
| |
| * auto scaling needs<code>docker-machine</code>
| |
| |
| |
| |-
| |
| |Kubernetes
| |
| |
| |
| * High elasticity/auto scaling
| |
| *separation of jobs by containers
| |
| *
| |
| |
| |
| * Additional Kubernetes needed (for security)
| |
| * Additional cluster needs maintenance
| |
| * More difficult to setup
| |
| |
| |
| * Could be used to strengthen Kubernetes knowledge
| |
| * Auto scaling needs elastic compute plattform
| |
| *Maybe a general purpose non-production cluster can be build?
| |
| |-
| |
| |OpenShift
| |
| |
| |
| |
| |
| * not in use in WMF
| |
| |
| |
| |}
| |
|
| |
| ==== Compute Environments ====
| |
| The following table compares the four main computing options for the GitLab Runner setup: WMCS, Ganeti, Bare Metal or Public Cloud.
| |
| {| class="wikitable"
| |
| |+
| |
| !Environment
| |
| !Advantages
| |
| !Disadvantages
| |
| !Additional considerations
| |
| |-
| |
| |WMCS
| |
| |
| |
| * High elasticity
| |
| * Kubernetes auto scaling can leverage OpenStack
| |
| |
| |
| * Only in Eqiad
| |
| *not fully trusted environment
| |
| |
| |
| * Elasticity is bound to appropriate quotas
| |
| * Kubernetes on OpenStack is new and different from existing Kubernetes solutions
| |
| |-
| |
| |Ganeti
| |
| |
| |
| * Trusted environment
| |
| |
| |
| * medium elasticity
| |
| |
| |
| |-
| |
| |Bare metal
| |
| |
| |
| * Trusted environment
| |
| * Similar environment to existing Kubernetes setups
| |
| |
| |
| * Low elasticity
| |
| * Machines have to be ordered and racked
| |
| |
| |
| * Could old/decommissioned machines be used as runners?
| |
| |-
| |
| |Public Cloud (e.g. GCP)
| |
| |
| |
| * High elasticity
| |
| * Low maintenance
| |
| * Easy Kubernetes setup (e.g. GKE)
| |
| |
| |
| * untrusted environment (see privacy section)
| |
| * Dependency to cloud provider
| |
| |
| |
| * Discussion about public cloud usage is needed
| |
| * Evaluation of privacy considerations is needed (see below)
| |
| |}
| |
|
| |
| ===== Elastic demand =====
| |
| Typically the demand of computing resources for building code and running test is not constant. The usage of CI peaks around African, European and US Timezones and workdays (see Grafana [https://grafana.wikimedia.org/d/000000283/zuul-job?orgId=1&from=now-7d&to=now dashboard] and [https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=contint2001&var-datasource=thanos&var-cluster=ci&from=now-7d&to=now dashboard]). So the ideal solution would adapt to this usage and scale computing resources up and down. This would maximize the utilization of resources and cover usage peaks. However this elasticity comes with costs. In general a dynamic provisioning of Runners is more complex than a static. Currently internal compute environments (such as Ganeti or Bare Metal) have limited elasticity, WMCS is somewhat elastic. So if high elasticity is needed, we have to consider using external providers like GKE. Which opens the discussion about privacy (see next chapter) and being independent from external parties.
| |
|
| |
| So even if a elastic Runner setup would be the better technical solution we have to ask if we really need high elasticity now.
| |
|
| |
| Further reading:
| |
|
| |
| https://docs.gitlab.com/runner/enterprise_guide/#autoscaling-configuration-one-or-more-runner-managers-multiple-workers
| |
|
| |
| https://docs.gitlab.com/runner/executors/kubernetes.html
| |
|
| |
| https://docs.gitlab.com/runner/configuration/autoscale.html
| |
|
| |
| ==== Privacy and trust considerations ====
| |
| Privacy is one core principal of WMF. So if public clouds are used we have to make sure this usage aligns with our privacy policy and doesn't cause any security risks.
| |
|
| |
| We have to think about what data is transmitted to public clouds during builds and tests. Do we include secrets, passwords or private user data when running a job? Do we need a special policy for CI variables and secrets? Do we consider this data leaked/compromised when transmitted to public cloud machines even when encrypted/restricted machines are used? We also have to think about how to secure the artifacts and test results of jobs running in public clouds. How do we implement trust? How do we check if artifacts (images, compiled code) or test results weren't compromised?
| |
|
| |
| The safest and easiest approach would be to implement two different Runner environments, one for untrusted builds and one for trusted builds. This solution was proposed bei ServiceOps<ref>https://people.wikimedia.org/~oblivian/ci/ci-threat.pdf</ref>.
| |
|
| |
| In GitLab terms this would mean hosting a [https://docs.gitlab.com/ee/ci/runners/runners_scope.html Shared Runner] for all untrusted projects and builds. This Shared Runners could be hosted in WMCS or a Public Cloud and if possible not inside the production network due to security considerations. Furthermore Specific Runners could be installed in a trusted environment and assigned to specific project. It is also possible to use this Specific Runners only for specific branches and tags, see [https://docs.gitlab.com/ee/ci/runners/configure_runners.html#prevent-runners-from-revealing-sensitive-information Protected Runners].
| |
|
| |
| ==== Monitoring of performance and usage ====
| |
| Gitlab-Runner support Prometheus metric export. This metrics and some Grafana dashboards should give insights in performance and usage. See [https://docs.gitlab.com/runner/monitoring/ Monitoring Gitlab Runner] documentation.
| |
|
| |
| However the Gitlab Runner exporter does not support authorization or https. So depending on where the Runners are hosted, a https proxy with authorization is required.
| |
|
| |
| <ref>https://docs.gitlab.com/runner/monitoring/#configuration-of-the-metrics-http-server</ref>
| |
|
| |
| ==== Open Questions: ====
| |
| Open questions:
| |
|
| |
| * Do we need high elasticity now/in the near future?
| |
| *Do we have any drawbacks with the current, non-elastic setup with static Jenkins machines?
| |
| * Do we want to open a discussion about public cloud usage and privacy?
| |
| * Do we want to be depended on public cloud offerings?
| |
| * Do we have the resources to plan, implement and maintain a additional Kubernetes Cluster?
| |
| * Is a GitLab Runner Kubernetes cluster the best way to accumulate Kubernetes knowledge?
| |
| *Is it possible to build docker images in Runners with the Docker Executor easily and securely?
| |
| *Can we secure the trusted Runners using a special set of Specific Runners and/or Protected Runners enough?
| |
| *Is it possible to have multiple [https://docs.gitlab.com/ee/administration/job_artifacts.html artifact stores] ([https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html and]) for different Runners?
| |
| <references /> | | <references /> |
