Heterogeneous deployment/Train deploys

Tuesday: New branch creation and deploy

Create the new branch in gerrit

The new branch can be created in Gerrit from anywhere. It is often faster to do this step on a host in the cluster to minimize the time needed to clone from gerrit.

The script is run as your regular user member of the wikidev group (as of Feb 16th 2016). You have to configure git:

git config --global user.name "FIRST NAME LAST NAME"
git config --global user.email "youremail@example.org"

Create a .netrc (on Windows _netrc) file in your home directory with the following content:

machine gerrit.wikimedia.org login myawesomeusername password MySuper/SecretPassword!

This username and password can obtained from Gerrit.

git clone https://gerrit.wikimedia.org:29418/mediawiki/tools/release.git
cd release/make-wmf-branch
./make-wmf-branch -n VERSION -o master
- (e.g. php make-wmf-branch -n 1.27.0-wmf.9 -o master)

Clone new branch

On tin, to create a new /srv/mediawiki-staging/php-VERSION directory run:
- /srv/mediawiki-staging/multiversion/checkoutMediaWiki VERSION php-VERSION
- (e.g. /srv/mediawiki-staging/multiversion/checkoutMediaWiki 1.27.0-wmf.9 php-1.27.0-wmf.9)

Apply security patches

Look for a patches directory at /srv/patches/VERSION. If one doesn't yet exist, copy over the previous version's directory. (e.g., cp -r /srv/patches/1.27.0-wmf.9 /srv/patches/1.27.0-wmf.10)
Patches should be named sequentially in the order that they will cleanly apply (e.g. 01-T123.patch, 02-T321.patch)
Check and apply each patch in both /srv/patches/VERSION/core and /srv/patches/VERSION/extensions/NAME to the new core checkout and extensions, respectively.
- You can check a core patch to see if it will apply cleanly with git apply --check --3way /srv/patches/VERSION/core/XX-TXXX.patch
- If the patch checks out, apply and commit it with git am --3way /srv/patches/VERSION/core/XX-TXXX.patch
- If the patch fails to apply, investigate whether it's due to a conflict (git status) or the patch having been merged since the new branch cut (search git log for the commit, etc.). If it turns out to be the latter, remove the patch file from the /srv/patches/VERSION directory. If you need extra help, contact Chad (ostriches) or Darian (dapatrick) in IRC (#mediawiki_security ^connect).

Create patches to update wikiversions.json

Create group0 to VERSION patch
- /srv/mediawiki-staging/multiversion/updateWikiversions group0 php-VERSION
  - (e.g. /srv/mediawiki-staging/multiversion/updateWikiversions group0 php-1.27.0-wmf.9)
- git add wikiversions.json
- DOLOGMSGNOLOG=1 git commit -m "Group0 to VERSION"
  - (e.g. DOLOGMSGNOLOG=1 git commit -m "Group0 to 1.27.0-wmf.9")

Send staged patches to gerrit for review

cd /srv/mediawiki-staging
git push origin HEAD:refs/for/master/VERSION
- (e.g. git push origin HEAD:refs/for/master/1.27.0-wmf.9)

Discard changes to working directory and index on tin

cd /srv/mediawiki-staging
DOLOGMSGNOLOG=1 git reset --hard origin/master

Clean up old stuff

Technically none of these steps are required, but you should do them anyway to make Chad happy :)

Remove clones of expired branches

For each branch that is more than 5 versions old, remove the clone from /srv/mediawiki-staging:

rm -rf /srv/mediawiki-staging/php-STALE_VERSION

Purge localization cache for now unused versions

scap-purge-l10n-cache --version VERSION-2
- (e.g. scap l10n-purge --version 1.28.0-wmf.10)

Remove left over files from expired branches

Because we don't rsync the localization CDB files during scap, some files will be left behind after scap runs for expired branches that were removed. Clean them up across the cluster using dsh

SSH_AUTH_SOCK=/run/keyholder/proxy.sock dsh -F 20 -M -g scap-masters -r ssh -o -oUser=mwdeploy -- rm -rf /srv/mediawiki/php-[VERSION]
SSH_AUTH_SOCK=/run/keyholder/proxy.sock dsh -F 20 -M -g scap-proxies -r ssh -o -oUser=mwdeploy -- rm -rf /srv/mediawiki/php-[VERSION]
SSH_AUTH_SOCK=/run/keyholder/proxy.sock dsh -F 20 -M -g mediawiki-installation -r ssh -o -oUser=mwdeploy -- rm -rf /srv/mediawiki/php-[VERSION]

To simplify this step, you can use the delete-stale-branch script which can be found in the Mediawiki Release Tools (rMREL) repository.

Sync to cluster and verify on testwiki

Edit /srv/mediawiki-staging/wikiversions.json and set testwiki to php-VERSION
- Do not commit and push to gerrit, only make this change locally on the deployment seerver
Run scap to (re)build localization caches and sync changes across the cluster
- scap sync "testwiki to php-VERSION and rebuild l10n cache"
Verify version change and l10n cache on testwiki
Revert local changes: git checkout -- wikiversions.json

Update deploy notes

Setup the tools you need to do this:

Clone mediawiki/tools/release.git
- git clone ssh://gerrit.wikimedia.org:29418/mediawiki/tools/release.git
Copy make-deploy-notes/auth.php.example to make-deploy-notes/auth.php and fill in your mediawiki.org useraname and password.
- $wiki->login( 'my user account', 'my awesome password' );
Clone mediawiki/core
- git clone ssh://gerrit.wikimedia.org:29418/mediawiki/core.git

For the new branch and any branches that may have changed in the last week:

Check out the branch locally
- git checkout wmf/VERSION
- (e.g. git checkout wmf/1.24wmf4)
If you don't already have the previous branch checked out, do that as well
- git checkout wmf/PREVIOUS-VERSION
- (e.g. git checkout wmf/1.24wmf3)
Update the submodules
- git submodule update --init --recursive
Run the change log generation and upload script
- php path/to/make-deploy-notes/uploadChangelog.php wmf/VERSION
  - (e.g. php path/to/make-deploy-notes/uploadChangelog.php wmf/1.24wmf4)
Repeat for additional branches as needed

Wait for deploy window

All of the changes above can be done at any time prior to the actual deployment window. Depending on how practiced you are and where you choose to run commands (full clones of mediawiki-core from outside the cluster can take a while) the steps up to this point will typically take 60 to 120 minutes.

Switch group0 wikis to VERSION

Review and submit group0 to VERSION patch in gerrit
Wait for gerrit/zuul/jenkins to merge the patch(es)
Pull patch(es) to deployment server
- cd /srv/mediawiki-staging
- git fetch
Check diff to ensure it is what you expect
- git diff HEAD..origin/master
Apply changes
- git rebase origin/master
Sync the change across the cluster
- scap sync-wikiversions "group0 to VERSION"
  - (e.g. scap sync-wikiversions "group0 to 1.24wmf4")
Verify that mediawikiwiki switched to the new version
Monitor irc and logstash and/or fatalmonitor for problems

Update roadmap

Change the Deployed to group (if you're using VisualEditor) or the 3rd parameter of the WMFReleaseTableRow template (if you're using the wikitext editor) to "0" (deployed to group0)n the branch roadmap.

Wednesday: group0 to group1 deploy

Clone the MediaWiki release tools to your home directory on tin
- ssh tin.eqiad.wmnet
- git clone https://gerrit.wikimedia.org/r/mediawiki/tools/release "$HOME/release"
Use the deploy-promote script inside ~/release/bin/ to update wikiversions.json
- ~/release/bin/deploy-promote
  - Promote group1 from VERSION-1 to VERSION? Enter to continue, ctl-c to cancel:
- The script automatically Code-Review +2 the patch in Gerrit. Once CI has merged it hit enter at the 2nd prompt:
  - Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions
- After the script run is complete, group1 wikis should be running VERSION

Thursday: group{0,1} to all deploy

Thursday deploy is very similar to the Wednesday deploy, the only difference in terms of procedure is the target group

Clone the MediaWiki release tools to your home directory on tin
Use ~/release/bin/deploy-promote all to update wikiversions.json
- The script automatically send to Gerrit and Code-Review +2 the patch. Once CI has merged it, hit enter at the 2nd prompt:
  - Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions
- After the script run is complete, all wikis should be running VERSION

Example Session:

you@laptop:~$ ssh deployment.eqiad.wmnet
you@tin:~$ git clone https://gerrit.wikimedia.org/r/mediawiki/tools/release "$HOME/release"
you@tin:~$  ./bin/deploy-promote all
Promote all from 1.28.0-wmf.15 to 1.28.0-wmf.16? [y/N] y
#!/usr/bin/env php
Updated /srv/mediawiki-staging/wikiversions.json: 0 inserted, 897 migrated.
/srv/mediawiki-staging/php is already up-to-date.
[master 67c2f0c] all wikis to 1.28.0-wmf.16
 1 file changed, 298 insertions(+), 298 deletions(-)
Counting objects: 21, done.
Delta compression using up to 6 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 2.12 KiB | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2)
remote: Processing changes: new: 1, refs: 1, done    
remote: 
remote: New Changes:
remote:   https://gerrit.wikimedia.org/r/306719 all wikis to 1.28.0-wmf.16
remote: 
To ssh://gerrit.wikimedia.org:29418/operations/mediawiki-config.git
 * [new branch]      HEAD -> refs/for/master/1.28.0-wmf.16%l=Code-Review+2
HEAD is now at ad6c345 [Beta Cluster] Remove PoolCounter override
Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions
[INFO] Running git pull
From https://gerrit.wikimedia.org/r/p/operations/mediawiki-config
   ad6c345..67c2f0c  master     -> origin/master
Updating ad6c345..67c2f0c
Fast-forward
 wikiversions.json | 596 ++--
 1 file changed, 298 insertions(+), 298 deletions(-)
[INFO] Running scap sync-wikiversions
19:01:53 Started sync-masters
sync-masters: 100% (ok: 1; fail: 0; left: 0)                                    
19:02:01 Finished sync-masters (duration: 00m 08s)
19:02:01 Started sync_wikiversions
19:02:01 Compiled /srv/mediawiki-staging/wikiversions.json to /srv/mediawiki-staging/wikiversions.php
sync_wikiversions: 100% (ok: 345; fail: 0; left: 0)                             
19:02:10 Finished sync_wikiversions (duration: 00m 09s)
19:02:10 rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.16
==================================================
Checking version on https://en.wikipedia.org/wiki/Special:Version
Expect: 1.28.0-wmf.16
Result: SUCCESS\n
==================================================

Heterogeneous deployment/Train deploys

Contents

Tuesday: New branch creation and deploy

Create the new branch in gerrit

Clone new branch

Apply security patches

Create patches to update wikiversions.json

Send staged patches to gerrit for review

Discard changes to working directory and index on tin

Clean up old stuff

Remove clones of expired branches

Purge localization cache for now unused versions

Remove left over files from expired branches

Sync to cluster and verify on testwiki

Update deploy notes

Wait for deploy window

Switch group0 wikis to VERSION

Update roadmap

Wednesday: group0 to group1 deploy

Thursday: group{0,1} to all deploy

Navigation menu

Heterogeneous deployment/Train deploys

Tuesday: New branch creation and deploy

Create the new branch in gerrit

Clone new branch

Apply security patches

Create patches to update wikiversions.json

Send staged patches to gerrit for review

Discard changes to working directory and index on tin

Clean up old stuff

Remove clones of expired branches

Purge localization cache for now unused versions

Remove left over files from expired branches

Sync to cluster and verify on testwiki

Update deploy notes

Wait for deploy window

Switch group0 wikis to VERSION

Update roadmap

Wednesday: group0 to group1 deploy

Thursday: group{0,1} to all deploy

Navigation menu

Search