You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Scap/Release: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Legoktm
imported>Clément Goubert
(Fix scap git repo URL)
 
(8 intermediate revisions by 5 users not shown)
Line 2: Line 2:


=== Building ===
=== Building ===
To prepare a new release of the Debian package for Scap, the '''Release Engineering Team''' needs to follow the steps in [https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/scap/+/master/RELEASE.md RELEASE.md] (in the Scap git repository), and then the '''SRE team''' needs to build and deploy the package, which are listed below.
To prepare a new release for Scap, the Release Engineering Team needs to follow the steps in [https://gitlab.wikimedia.org/repos/releng/scap/-/blob/master/RELEASE.md RELEASE.md] (in the Scap git repository)


The Debian package for scap can be built with <code>git-buildpackage</code>. More specifically: for production the standard procedure is to have a Debian source package built on the <tt>package_builder</tt> machine.
===Roll out to production===
Performed by a member of the Release Engineering Team:


  git clone <nowiki>https://gerrit.wikimedia.org/r/mediawiki/tools/scap</nowiki>
*Make sure you select a time window where no deployments are happening
  cd scap
*On the main deployment server ('''deploy1002.eqiad.wmnet''' at the time of writing), run '''<code>scap install-world</code>'''. This will select for installation the latest available version tag from the git repo and prompt you for confirmation.
  V=4.0.0 # Change value as appropriate
*Log into '''<code>mwdebug*</code>''' servers and check there if a '''<code>scap pull</code>''' still works
  git checkout -b $V $V
* Check if scap still works for the scap3 code path, and on stretch. You will need to run a test deployment to test environment. Run the following from the main deployment server:<syntaxhighlight lang="bash">
  WIKIMEDIA=yes gbp buildpackage -sa -us -uc --git-pbuilder --git-no-pbuilder-autoconf --git-dist=buster --git-debian-branch=$V
cd /srv/deployment/restbase/deploy/; scap deploy --environment dev-cluster
...
canary deploy successful. Continue? [y]es/[n]o/[c]ontinue all groups: n
Rollback all deployed groups? [Y/n]: Y
</syntaxhighlight>


=== Uploading to apt repos ===
===Rollback===
 
In case a rollback is necessary, it is possible to pass a previous version tag to the install command. For example: '''<code>scap install-world --version 4.9.3</code>'''
The resulting package will be in <tt>/var/cache/pbuilder/result/stretch-amd64/</tt> and needs to be uploaded to the apt repo (e.g. from <tt>apt1001.wikimedia.org</tt>)
===Target host bootstrapping===
 
Hosts added to the list of targets should be automatically primed with the latest deployed scap version. But if the automated process failed for a host for whatever reason, it is possible to trigger it manually from the main deployment server with: '''<code>scap install-world --batch</code>''' This will ensure all hosts are in sync with the version staged in the deployment server (i.e. the last deployed scap version)
  export DIST="buster"
  rsync -vaz deneb.codfw.wmnet::pbuilder-result/$DIST-amd64/*scap* deb/
  sudo -i reprepro --ignore=wrongdistribution include $DIST-wikimedia $(pwd)/deb/scap_<VERSION>_amd64.changes
  # scap is compatible as-is with stretch and buster, just copy the package there
  sudo -i reprepro copy stretch-wikimedia $DIST-wikimedia scap
 
If you get errors about a missing GPG key and therefore not exporting indices you need to import the right key like described on [[Reprepro#Adding_a_new_external_repository]].
 
If signing still fails after you imported the key you need to ensure it looks for them in the right home dir, see [[Reprepro#If_signing_fails]].
 
If this already happened and you can't get it to export indices by repeating the "include" command you can work around it by copying it back from another distro (reprepro copy stretch-wikimedia buster-wikimedia scap).
 
Not having exported indices will manifest as "reprepro ls shows the right version but it is not found on a client even after running apt-get update".
 
=== Roll out to production ===
You will then need to use [[Software_deployment#Deploying_a_software_update_fleet-wide_using_debdeploy_and_debmonitor|Debdeploy]] to deploy the new package in production.
 
* If you just uploaded the package, use cumin to run <code>apt-get update</code> on the hosts or wait until puppet will do it automatically
*Start by deploying to <code>mw-canary</code>, <code>mw-jobrunner-canary</code>, <code>parsoid-canary</code> (cumin aliases), then login to '''<code>mwdebug*</code>''' servers and check there if a '''<code>scap pull</code>''' still works.
* Later or the next day you can rollout to <code>all</code>.
 
=== Resources ===
There are Debian [https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Standards-Version Versioning guidelines]

Latest revision as of 10:35, 9 November 2022

This page is documentation for releasing a new version of Scap to Wikimedia Production.

Building

To prepare a new release for Scap, the Release Engineering Team needs to follow the steps in RELEASE.md (in the Scap git repository)

Roll out to production

Performed by a member of the Release Engineering Team:

  • Make sure you select a time window where no deployments are happening
  • On the main deployment server (deploy1002.eqiad.wmnet at the time of writing), run scap install-world. This will select for installation the latest available version tag from the git repo and prompt you for confirmation.
  • Log into mwdebug* servers and check there if a scap pull still works
  • Check if scap still works for the scap3 code path, and on stretch. You will need to run a test deployment to test environment. Run the following from the main deployment server:
    cd /srv/deployment/restbase/deploy/; scap deploy --environment dev-cluster
    ...
    canary deploy successful. Continue? [y]es/[n]o/[c]ontinue all groups: n
    Rollback all deployed groups? [Y/n]: Y
    

Rollback

In case a rollback is necessary, it is possible to pass a previous version tag to the install command. For example: scap install-world --version 4.9.3

Target host bootstrapping

Hosts added to the list of targets should be automatically primed with the latest deployed scap version. But if the automated process failed for a host for whatever reason, it is possible to trigger it manually from the main deployment server with: scap install-world --batch This will ensure all hosts are in sync with the version staged in the deployment server (i.e. the last deployed scap version)