You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

GitLab/Upgrade: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Jelto
(update upgrade cadence)
imported>Jelto
No edit summary
Line 27: Line 27:
* minor upgrades (issued monthly): as needed for upstream features and bugfixes, if possible also monthly
* minor upgrades (issued monthly): as needed for upstream features and bugfixes, if possible also monthly
* major upgrades (issued annually): once per year, after release has had time to be tested by early adopters and receive fixes
* major upgrades (issued annually): once per year, after release has had time to be tested by early adopters and receive fixes
==== Upgrade GitLab Debian package ====
* Update <code>modules/aptrepo/files/updates</code> in <code>operations/puppet/</code> similar to [[gerrit:c/operations/puppet/+/774905|774905]]. Set hook commands <code>--eq</code> to desired version
* Get +1 for aptrepo change and merge
* run puppet on <code>apt1001</code>
* Run <code>reprepro --component thirdparty/gitlab checkupdate buster-wikimedia</code> on <code>apt1001</code> to check if new version is found
* Run <code>reprepro --component thirdparty/gitlab update buster-wikimedia</code> on <code>apt1001</code> to update package
* repeat for <code>thirdparty/gitlab-runner</code> if needed
* read [[Reprepro#Updating external repositories]] for more information


==== Prerequisites ====
==== Prerequisites ====
Line 32: Line 42:
* Make yourself familiar with [https://docs.gitlab.com/ee/policy/maintenance.html GitLab release and maintenance policy]
* Make yourself familiar with [https://docs.gitlab.com/ee/policy/maintenance.html GitLab release and maintenance policy]
* Select exact GitLab version to upgrade to and find if it's a major, minor or patch upgrade
* Select exact GitLab version to upgrade to and find if it's a major, minor or patch upgrade
* Read release notes of all versions between current and selected one
* Read [https://about.gitlab.com/releases/categories/releases/ release notes] of all versions between current and selected one
* Determine the proper upgrade path. Make sure an upgrade from the current to the new version is possible. Never upgrade over two major versions in a single step, that will (with great probability) lead to a broken installation
* Determine the proper upgrade path. Make sure an upgrade from the current to the new version is possible. Never upgrade over two major versions in a single step, that will (with great probability) lead to a broken installation
* Make sure the package with the new version is present in [https://debmonitor.wikimedia.org/packages/gitlab-ce debmonitor]
* Make sure the package with the new version is present in [https://debmonitor.wikimedia.org/packages/gitlab-ce debmonitor]
Line 43: Line 53:
</syntaxhighlight>
</syntaxhighlight>


* disable [[gerrit:c/operations/puppet/+/744768|automatic restore on the replica]]
* preload to-be-installed GitLab CE packages before upgrading:
*preload to-be-installed GitLab CE packages before upgrading:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
sudo apt-get update && sudo apt-get install gitlab-ce=14.0.10-ce.0 --download-only
sudo apt-get update && sudo apt-get install gitlab-ce=14.0.10-ce.0 --download-only
Line 57: Line 66:
==== Upgrading GitLab ====
==== Upgrading GitLab ====


* Downtime GitLab host in Icinga:
* Downtime GitLab host in Icinga if long migration is expected (see release notes, not needed in most cases):
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
sudo cookbook sre.hosts.downtime -r "upgrade <gitlab_host> to new version https://phabricator.wikmiedia.org/<ID>" -H 1 <gitlab_host>
sudo cookbook sre.hosts.downtime -r "upgrade <gitlab_host> to new version https://phabricator.wikmiedia.org/<ID>" -H 1 <gitlab_host>

Revision as of 15:56, 29 March 2022

WMF GitLab is installed via Debian package named gitlab-ce. This is called an Omnibus setup. So to upgrade GitLab, this package is upgraded to a newer version. The upgrade process is described here. For more detail, please see the GitLab upgrade documentation here and the Omnibus specific upgrade documentation here.

WMF upgrade path

The upgrade of GitLab should be applied in the following order:

  1. upgrade gitlab-prod-1001 machine in WMCS/horizon, project devtools
  2. upgrade GitLab replica (currently gitlab2001, please check)
  3. upgrade production GitLab (currently gitlab1001, please check)

Recommended upgrade cadence:

  • security patches: as soon as possible
  • minor upgrades (issued monthly): as needed for upstream features and bugfixes, if possible also monthly
  • major upgrades (issued annually): once per year, after release has had time to be tested by early adopters and receive fixes

Upgrade GitLab Debian package

  • Update modules/aptrepo/files/updates in operations/puppet/ similar to 774905. Set hook commands --eq to desired version
  • Get +1 for aptrepo change and merge
  • run puppet on apt1001
  • Run reprepro --component thirdparty/gitlab checkupdate buster-wikimedia on apt1001 to check if new version is found
  • Run reprepro --component thirdparty/gitlab update buster-wikimedia on apt1001 to update package
  • repeat for thirdparty/gitlab-runner if needed
  • read Reprepro#Updating external repositories for more information

Prerequisites

  • Make yourself familiar with GitLab release and maintenance policy
  • Select exact GitLab version to upgrade to and find if it's a major, minor or patch upgrade
  • Read release notes of all versions between current and selected one
  • Determine the proper upgrade path. Make sure an upgrade from the current to the new version is possible. Never upgrade over two major versions in a single step, that will (with great probability) lead to a broken installation
  • Make sure the package with the new version is present in debmonitor
  • Determine if any manual migrations are required in your upgrade; built-in PostgreSQL database server upgrade may be required between major versions
  • make both full GitLab data and full configuration data backups before upgrading:
sudo /usr/bin/gitlab-backup create CRON=1 STRATEGY=copy GZIP_RSYNCABLE=yes SKIP=builds,artifacts,registry GITLAB_BACKUP_MAX_CONCURRENCY=4 GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY=1
sudo /usr/bin/gitlab-ctl backup-etc
  • preload to-be-installed GitLab CE packages before upgrading:
sudo apt-get update && sudo apt-get install gitlab-ce=14.0.10-ce.0 --download-only
  • if you have GitLab Runners connected to your GitLab Server, it is recommended to pause all runners and wait until all jobs are finished before starting the upgrade. TODO: Automate/script this step?
  • Check if any background migrations are running:
sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

Upgrading GitLab

  • Downtime GitLab host in Icinga if long migration is expected (see release notes, not needed in most cases):
sudo cookbook sre.hosts.downtime -r "upgrade <gitlab_host> to new version https://phabricator.wikmiedia.org/<ID>" -H 1 <gitlab_host>
  • Run required manual configuration steps if needed (see release notes, not needed in most cases)
  • Install new GitLab version:
sudo apt-get install gitlab-ce=14.0.10-ce.0
  • Wait for GitLab to finish all migrations and restarts

Upgrading GitLab without downtime

WIP: may be used in the future. See zero downtime upgrades.

Upgrading GitLab shared runners

Shared runners are currently in the gitlab-runners project in WMCS and in codfw and eqiad (gitlab-runner100* and gitlab-runner200*).

sudo apt-get update && sudo apt-get install gitlab-runner
  • Runners will restart automatically. If needed, restart runners:
sudo /usr/bin/gitlab-runner restart

Steps after upgrading

  • Run required manual steps/migrations (see release notes. not needed in most cases)
  • Check that all background migrations are fully finished and background migration queue is empty
  • Make sure that GitLab is up and running after upgrade; please give it several minutes to calm down
  • Check monitoring systems of GitLab (especially Icinga alerts)
  • Run basic smoke tests (make sure that web UI works, authentication works, ssh cloning works)
  • Re-enable paused runners in the GitLab admin interface at https://gitlab.wikimedia.org/admin/runners
  • Re-enable restore for replica