Heterogeneous deployment/Train deploys

New branch creation and deploy

Create the new branch in gerrit

The new branch can be created in gerrit from anywhere. It is often faster to do this step on a host in the cluster to minimize the time needed to clone from gerrit.

For now, make-wmf-branch requires a newer version of php than is on the deploy host (tin), so running this on your laptop is required. See this comment in phab

• git clone ssh://gerrit.wikimedia.org:29418/mediawiki/tools/release.git
• cd release/make-wmf-branch
• ./make-wmf-branch -n VERSION -o master
  • (e.g. php make-wmf-branch -n 1.27.0-wmf.9 -o master)

Remove symlinks to expired branches

Any branch checkout on the deployment server that has not been used for more than 5 weeks can be safely removed to reduce disk usage across the cluster.

• /srv/mediawiki-staging/multiversion/deleteMediaWiki STALE_VERSION
• git rm -r docroot/bits/static-STALE_VERSION
• git rm -r w/static-STALE_VERSION
• DOLOGMSGNOLOG=1 git commit -m "Remove STALE_VERSION symlinks"

Clone new branch

• On tin, to create a new /srv/mediawiki-staging/php-VERSION directory run:
  • /srv/mediawiki-staging/multiversion/checkoutMediaWiki VERSION php-VERSION
  • (e.g. /srv/mediawiki-staging/multiversion/checkoutMediaWiki 1.27.0-wmf.9 php-1.27.0-wmf.9)

Apply security patches

• Check /srv/patches/core/*.patch for any patches for /srv/mediawiki-staging/php-VERSION
• Check /srv/patches/extensions/EXTENSION-NAME/*.patch for any patches for /srv/mediawiki-staging/php-VERSION/extensions/EXTENSION
• Can use git apply --check PATCH_FILE to check if patches apply cleanly (e.g., cd /srv/mediawiki-staging/1.27.0-wmf.9; git apply --check /srv/patches/core/TXXXXXX.patch
• Some patches may require git apply --check --3way to apply cleanly
• If any patch doesn't apply cleanly, contact Chad (ostriches) or Chris (csteipp) in IRC.
• There is never a patch that shouldn't apply in /srv/patches if that patch has merged it should be removed.

Create symlink patch for common/docroot

• Create symlinks patch for changes made in common/docroot and common/w by checkoutMediaWiki
  • cd /srv/mediawiki-staging
  • git add docroot/bits/static-VERSION w/static-VERSION
  • DOLOGMSGNOLOG=1 git commit -m "Add VERSION symlinks"

Create patches to update wikiversions.json

• Create group0 to VERSION patch
  • /srv/mediawiki-staging/multiversion/updateWikiversions group0 php-VERSION
    • (e.g. /srv/mediawiki-staging/multiversion/updateWikiversions group0 php-1.27.0-wmf.9)
  • git add wikiversions.json
  • DOLOGMSGNOLOG=1 git commit -m "Group0 to VERSION"
    • (e.g. DOLOGMSGNOLOG=1 git commit -m "Group0 to 1.27.0-wmf.9")

Send staged patches to gerrit for review

• cd /srv/mediawiki-staging
• git push origin HEAD:refs/for/master/VERSION
  • (e.g. git push origin HEAD:refs/for/master/1.27.0-wmf.9)

Discard changes to working directory and index on tin

• cd /srv/mediawiki-staging
• DOLOGMSGNOLOG=1 git reset --hard origin/master

Restore symlinks on deployment server

• Review and submit symlinks patches in gerrit
  • This includes the patches for any expired branches that were removed and the new branch that was added.
• Wait for gerrit/zuul/jenkins to merge the patch(es)
• Pull patch(es) to deployment server
  • cd /srv/mediawiki-staging
  • git fetch
• Check diff to ensure it is what you expect
  • git diff HEAD..origin/master
• Apply changes
  • git rebase origin/master

Remove clones of expired branches

For each branch that you removed symlinks for, remove the clone from /srv/mediawiki-staging:

• rm -rf /srv/mediawiki-staging/php-STALE_VERSION

Sync to cluster and verify on testwiki

• Edit /srv/mediawiki-staging/wikiversions.json and set testwiki to php-VERSION
  • Do not commit and push to gerrit, only make this change locally on the deployment seerver
• Run scap to (re)build localization caches and sync changes across the cluster
  • scap "testwiki to php-VERSION and rebuild l10n cache"
• Verify version change and l10n cache on testwiki
• Revert local changes: git checkout -- wikiversions.json

Remove left over files from expired branches

Because we don't rsync the localization CDB files during scap, some files will be left behind after scap runs for expired branches that were removed. Clean them up across the cluster using dsh

• SSH_AUTH_SOCK=/run/keyholder/proxy.sock dsh -F 20 -M -g mediawiki-installation -r ssh -o -oUser=mwdeploy -- rm -rf /srv/mediawiki/php-STALE_VERSION

Wait for deploy window

All of the changes above can be done at any time prior to the actual deployment window. Depending on how practiced you are and where you choose to run commands (full clones of mediawiki-core from outside the cluster can take a while) the steps up to this point will typically take 60 to 120 minutes.

Switch all wikis to VERSION-1

• Review and submit all to VERSION-1 patch in gerrit
• Wait for gerrit/zuul/jenkins to merge the patch(es)
• Pull patch(es) to deployment server
  • cd /srv/mediawiki-staging
  • git fetch
• Check diff to ensure it is what you expect
  • git diff HEAD..origin/master
• Apply changes
  • git rebase origin/master
• Sync the change across the cluster
  • sync-wikiversions "wikipedias to VERSION-1"
    • (e.g. sync-wikiversions "wikipedias to 1.24wmf3")
• Verify that enwiki switched to the new version
• Monitor irc and logstash and/or fatalmonitor for problems

Switch group0 wikis to VERSION

• Review and submit group0 to VERSION patch in gerrit
• Wait for gerrit/zuul/jenkins to merge the patch(es)
• Pull patch(es) to deployment server
  • cd /srv/mediawiki-staging
  • git fetch
• Check diff to ensure it is what you expect
  • git diff HEAD..origin/master
• Apply changes
  • git rebase origin/master
• Sync the change across the cluster
  • sync-wikiversions "group0 to VERSION"
    • (e.g. sync-wikiversions "group0 to 1.24wmf4")
• Verify that mediawikiwiki switched to the new version
• Monitor irc and logstash and/or fatalmonitor for problems

Purge localization cache for now unused versions

• scap-purge-l10n-cache --version VERSION-2
  • (e.g. scap-purge-l10n-cache --version 1.24wmf2)

Update deploy notes

Setup the tools you need to do this:

• Clone mediawiki/tools/release.git
  • git clone ssh://gerrit.wikimedia.org:29418/mediawiki/tools/release.git
• Add wiki credentials to make-deploy-notes/uploadChangelog.php
  • $wiki->login( 'my user account', 'my awesome password' );
• Clone mediawiki/core
  • git clone ssh://gerrit.wikimedia.org:29418/mediawiki/core.git

For the new branch and any branches that may have changed in the last week:

• Check out the branch locally
  • git checkout wmf/VERSION
  • (e.g. git checkout wmf/1.24wmf4)
• Update the submodules
  • git submodule update --init --recursive
• Run the change log generation and upload script
  • php path/to/make-deploy-notes/uploadChangelog.php wmf/VERSION
    • (e.g. php path/to/make-deploy-notes/uploadChangelog.php wmf/1.24wmf4)
• Repeat for additional branches as needed

Update roadmap

• Add {{done}} to the appropriate rows on the branch roadmap.

group0 to group1 deploy

1. Create group1 wikis to VERSION patch

• /srv/mediawiki-staging/multiversion/updateWikiversions group1 php-${VERSION-1}
  • (e.g. /srv/mediawiki-staging/multiversion/updateWikiversions group1 php-1.24wmf3)
• git add wikiversions.json
• DOLOGMSGNOLOG=1 git commit -m 'Group1 wikis to ${VERSION}'
  • (e.g. DOLOGMSGNOLOG=1 git commit -m 'Group1 wikis to 1.24wmf4')

2. Send staged patch to gerrit for review

• cd /srv/mediawiki-staging
• git push origin HEAD:refs/for/master/VERSION
  • (e.g. git push origin HEAD:refs/for/master/1.24wmf4)

3. Discard local changes

• git reset --hard HEAD^

4. Switch group1 wikis to VERSION

• Review and submit group1 to VERSION patch in gerrit
• Wait for gerrit/zuul/jenkins to merge the patch(es)
• Pull patch(es) to deployment server
  • cd /srv/mediawiki-staging
  • git fetch
• Check diff to ensure it is what you expect
  • git diff HEAD..origin/master
• Apply changes
  • git rebase origin/master
• Sync the change across the cluster
  • sync-wikiversions 'group1 to VERSION'
    • (e.g. sync-wikiversions 'group1 to 1.24wmf4')
• Verify that enwikinews switched to the new version
• Monitor irc and logstash and/or fatalmonitor for problems

Automation

The following script will automate steps 1. and 2.

#!/bin/bash

set -e

cd /srv/mediawiki-staging/
OLDVERSION=`/srv/mediawiki-staging/multiversion/activeMWVersions|cut -d " " -f1`
NEWVERSION=`/srv/mediawiki-staging/multiversion/activeMWVersions|cut -d " " -f2`
/srv/mediawiki-staging/multiversion/updateWikiversions all php-${NEWVERSION}
/srv/mediawiki-staging/multiversion/updateWikiversions wikipedia php-${OLDVERSION}
git add wikiversions.json
DOLOGMSGNOLOG=1 git commit -m "Group1 wikis to ${NEWVERSION}"
git push origin HEAD:refs/for/master/${NEWVERSION}