You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Scap/Release: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Elukey
imported>Jaime Nuche
(Edited for clarity)
Line 2: Line 2:


=== Building ===
=== Building ===
To prepare a new release of the Debian package for Scap, the '''Release Engineering Team''' needs to follow the steps in [https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/scap/+/master/RELEASE.md RELEASE.md] (in the Scap git repository), and then the '''SRE team''' needs to build and deploy the package, which are listed below.
To prepare a new release for Scap, the Release Engineering Team needs to follow the steps in [https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/scap/+/master/RELEASE.md RELEASE.md] (in the Scap git repository)


The Debian package for scap can be built with <code>git-buildpackage</code>. More specifically: for production the standard procedure is to have a Debian source package built on the <tt>package_builder</tt> machine (deneb).
===Roll out to production===
Performed by a member of the Release Engineering Team:


  git clone <nowiki>https://gerrit.wikimedia.org/r/mediawiki/tools/scap</nowiki>
*Make sure you select a time window where no deployments are happening
  cd scap
*On the main deployment server ('''deploy1002.eqiad.wmnet''' at the time of writing), run '''<code>scap install-world</code>'''. This will select for installation the latest available version tag from the git repo and prompt you for confirmation.
  V=4.0.0 # Change value as appropriate
*Log into '''<code>mwdebug*</code>''' servers and check there if a '''<code>scap pull</code>''' still works
  git checkout -b $V $V
* Check if scap still works for the scap3 code path, and on stretch. You will need to run a test deployment to test environment. Run the following from the main deployment server:<syntaxhighlight lang="bash">
  WIKIMEDIA=yes gbp buildpackage -sa -us -uc --git-pbuilder --git-no-pbuilder-autoconf --git-dist=buster --git-debian-branch=$V
 
=== Uploading to apt repos ===
 
The resulting package will be in <tt>/var/cache/pbuilder/result/buster-amd64/</tt> and needs to be uploaded to the apt repo (e.g. from <tt>apt1001.wikimedia.org</tt>)
 
  export DIST="buster"
  V=4.0.0 # Change as appropriate
  rsync -vaz build2001.codfw.wmnet::pbuilder-result/$DIST-amd64/*scap*$V* deb/
  sudo -i reprepro --ignore=wrongdistribution include $DIST-wikimedia $(pwd)/deb/scap_${V}-1_amd64.changes
  # scap is compatible as-is with stretch and bullseye, just copy the package there
  sudo -i reprepro copy stretch-wikimedia $DIST-wikimedia scap
  sudo -i reprepro copy bullseye-wikimedia $DIST-wikimedia scap
 
If you get errors about a missing GPG key and therefore not exporting indices you need to import the right key like described on [[Reprepro#Adding_a_new_external_repository]].
 
If signing still fails after you imported the key you need to ensure it looks for them in the right home dir, see [[Reprepro#If_signing_fails]].
 
If this already happened and you can't get it to export indices by repeating the "include" command you can work around it by copying it back from another distro (reprepro copy stretch-wikimedia buster-wikimedia scap).
 
Not having exported indices will manifest as "reprepro ls shows the right version but it is not found on a client even after running apt-get update".
 
=== Roll out to production ===
You will then need to use [[Software_deployment#Deploying_a_software_update_fleet-wide_using_debdeploy_and_debmonitor|Debdeploy]] to deploy the new package in production.
 
* If you just uploaded the package, use cumin to run <code>apt-get update</code> on the hosts or wait until puppet will do it automatically
*Generate a debdeploy-spec using <code>generate-debdeploy-spec</code> on cumin:
**name: <code>scap</code>
**comment: optional or task ID
**update type: <code>tool</code>
**fixed versions in bullseye, buster and stretch: <code>4.0.0-1</code> (change as appropriate, use Debian naming schema)
**skip renaming
*Start by deploying to <code>mw-canary</code>, <code>mw-jobrunner-canary</code>, <code>parsoid-canary</code> (on cumin using following aliases)<syntaxhighlight lang="bash">
sudo debdeploy deploy -Q 'A:mw-canary or A:parsoid-canary or A:mw-jobrunner-canary' -u $(date '+%Y-%m-%d-scap.yaml')
</syntaxhighlight>
*login to '''<code>mwdebug*</code>''' servers and check there if a '''<code>scap pull</code>''' still works  
 
*Also deploy to <code>restbase-canary</code> (on cumin using following aliases):<syntaxhighlight lang="bash">
sudo debdeploy deploy -Q 'A:restbase-canary' -u $(date '+%Y-%m-%d-scap.yaml')
</syntaxhighlight>
*Check if scap still works for the scap3 code path, and on stretch. You will need to run a test deployment to test environment. Run the following deploy from deploy hosts:<syntaxhighlight lang="bash">
cd /srv/deployment/restbase/deploy/; scap deploy --environment dev-cluster
cd /srv/deployment/restbase/deploy/; scap deploy --environment dev-cluster
...
...
Line 56: Line 16:
Rollback all deployed groups? [Y/n]: Y
Rollback all deployed groups? [Y/n]: Y
</syntaxhighlight>
</syntaxhighlight>
*Later or the next day you can rollout to <code>all</code> (on cumin).<syntaxhighlight lang="bash">
sudo debdeploy deploy -s all -u $(date '+%Y-%m-%d-scap.yaml')
</syntaxhighlight>


===Resources===
===Rollback===
There are Debian [https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Standards-Version Versioning guidelines]
In case a rollback is necessary, it is possible to pass a previous version tag to the install command. For example: '''<code>scap install-world --version 4.9.3</code>'''
===Target host bootstrapping===
Hosts added to the list of targets should be automatically primed with the latest deployed scap version. But if the automated process failed for a host for whatever reason, it is possible to trigger it manually from the main deployment server with: '''<code>scap install-world --batch</code>''' This will ensure all hosts are in sync with the version staged in the deployment server (i.e. the last deployed scap version)

Revision as of 15:08, 15 June 2022

This page is documentation for releasing a new version of Scap to Wikimedia Production.

Building

To prepare a new release for Scap, the Release Engineering Team needs to follow the steps in RELEASE.md (in the Scap git repository)

Roll out to production

Performed by a member of the Release Engineering Team:

  • Make sure you select a time window where no deployments are happening
  • On the main deployment server (deploy1002.eqiad.wmnet at the time of writing), run scap install-world. This will select for installation the latest available version tag from the git repo and prompt you for confirmation.
  • Log into mwdebug* servers and check there if a scap pull still works
  • Check if scap still works for the scap3 code path, and on stretch. You will need to run a test deployment to test environment. Run the following from the main deployment server:
    cd /srv/deployment/restbase/deploy/; scap deploy --environment dev-cluster
...
canary deploy successful. Continue? [y]es/[n]o/[c]ontinue all groups: n
Rollback all deployed groups? [Y/n]: Y

Rollback

In case a rollback is necessary, it is possible to pass a previous version tag to the install command. For example: scap install-world --version 4.9.3

Target host bootstrapping

Hosts added to the list of targets should be automatically primed with the latest deployed scap version. But if the automated process failed for a host for whatever reason, it is possible to trigger it manually from the main deployment server with: scap install-world --batch This will ensure all hosts are in sync with the version staged in the deployment server (i.e. the last deployed scap version)

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Scap/Release&oldid=591366"