Backport windows/Deployers

Deployments
Jump to current event Deployment calendar Long-term calendar 1.38 Roadmap Train overview Archive Blocking tasks (Current task) Holding the train 🚨 Emergencies How to deploy code Heterogeneous deployment Train deploys Backport windows How to deploy backports Backport vs Train vs Special Window Train vs Backport Special Windows Backport windows

This document is intended to provide detailed instructions for deployers as to how to run the backport windows. Hopefully, this document will prove useful to new deployers as well as provide a place for more experienced deployers to take notes on any tips and tricks they have discovered in the course of doing deployments.

General advice

• Claim the window early to avoid confusion.

When jouncebot pings deployers in #wikimedia-operations ^connect, if you want to run that window, say so I can do the deploys today!

• Try to think out loud and be explicit.

If you are nervous about deploying a particular patch, mention it to the patch owner. It's better to have a conversation than to quietly fret over patches

• Be prepared.

Open all your SSH connections and error logs before you start deploying code.

• Release the window early when done to mark the rest of the window free.

!log EU deploys done, !log Morning deploys done, !log Evening deploys done, or something to that effect.

git

• git can be confusing. It is more confusing when you're trying to accomplish something under pressure. There are a few config options that might be helpful.

1. Use a git-aware prompt.
   The git contrib git-prompt script is available on deployment servers. There are instructions for use in comments at the beginning of the file. One simple way to use it is to add the following to your ~/.profile:

   GIT_PS1_SHOWUNTRACKEDFILES=1
   GIT_PS1_SHOWDIRTYSTATE=1
   GIT_PS1_SHOWUPSTREAM="auto verbose"
   . /etc/bash_completion.d/git-prompt
   PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '
   

2. status.submoduleSummary
   Submodules have limited visibility in git status and it's easy to miss the git submodule update step. After adding status.submoduleSummary to your ~/.gitconfig git will show you a short summary of submodule changes in git status. Set it by executing:

   you@deploy1002:~$ git config --global status.submoduleSummary true
   

SSH Connections and Error Logs

When running the window, it is helpful to watch error logs as you deploy so that you can be sure nothing you have just deployed is broken. Also, there are several machines on which you may need to run commands depending on the nature of the deploy; it's good to open all SSH connections before you have to think about them.

A script to set up these browser tabs and terminals automatically is available.

Browser tabs

• The Deployments calendar
• Logstash MediaWiki Errors
• Logstash MediaWiki Javascript errors
• Logstash Event log (mwdebug1002)
• Zuul Dashboard
• Wikimedia MediaWiki versions Dashboard
• Also, ensure you're signed in on Gerrit

Terminal tabs

• mwlog1001.eqiad.wmnet - to run logspam-watch, which you may wish to keep an eye during the window.
  • You may also run logspam for a one-off listing of recent errors, suitable for grepping.
• deployment.eqiad.wmnet (which is an alias to a deployment host in the currently primary data centre) - This is where you stage changes.
• mwdebug1002 - This is where you'll run scap pull to verify changes with X-Wikimedia-Debug.
• mwmaint1002.eqiad.wmnet or mwmaint2001.codfw.wmnet - To run maintenance scripts (if needed).

you@laptop$ ssh mwlog1001.eqiad.wmnet

you@mwlog1001:~$ logspam-watch

you@laptop$ ssh deployment.eqiad.wmnet

you@deploy1002:~$ cd /srv/mediawiki-staging

you@laptop$ ssh mwdebug1002.eqiad.wmnet

you@mwdebug1002:~$ scap pull

you@laptop$ ssh mwmaint1002.eqiad.wmnet

you@mwmaint1002:~$

Merging and applying patches

Note that there is a tool which, for any given gerrit change, produces a list of commands for deployment and revert that can be copy-pasted, and this tool is automatically linked to patches added to the Deployment Calendar via Module:Gerrit.

You will be merging patches either for a wmf branch of mediawiki repositories (core, or a WMF-installed extension or skin), or the operations/mediawiki-config repo.

Merging patches

When +2ing patches, it's often helpful to have the Zuul Dashboard open

• to ensure that Zuul is picking up your changes,
• to see how long (approximately) the test will take before it auto-merges.

It's a good practice to put Backport as the comment when you +2 before you click Publish Comments to ensure that there is a record of why you approved the patch.

Fetching patches

After the patch has merged in Gerrit, you need to fetch it down to deployment.eqiad.wmnet. Make sure that the code you fetch down to deployment.eqiad.wmnet is the code you expected to fetch down.

Use git log -p HEAD..@{u} after you git fetch to check that the patch(es) you fetched down were the same ones the you +2'd. If they aren't, poke the person that wrote the patch in #wikimedia-operations ^connect to figure out what to do with the fetched code. It's always better to ask than to do something silently and unilaterally.

Process is slightly different for operations/mediawiki-config, mediawiki/core, mediawiki/extensions and mediawiki/skins.

operations/mediawiki-config

you@deploy1002:/srv/mediawiki-staging$ git status

you@deploy1002:/srv/mediawiki-staging$ git fetch

you@deploy1002:/srv/mediawiki-staging$ git log -p HEAD..@{u}

you@deploy1002:/srv/mediawiki-staging$ git rebase

mediawiki/core

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git status

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git fetch

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git log HEAD..@{u}

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git rebase

mediawiki/extensions and mediawiki/skins

As soon as the change to an extension or skin gets merged, Gerrit bumps the associated submodule in the wmf/* branch of mediawiki/core. To deploy, you thus just have to fetch that parent repository, verify the difference between the current state (HEAD) and the tracked remote branch (@{u}), rebase and update the submodule:

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git status

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git fetch

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git log -p HEAD..@{u}

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git rebase

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git status [extensions|skins]/[NAME]

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git submodule update [extensions|skins]/[NAME]

Security Patches

Refer to How to deploy code -> Security patches for information about security patches.

Deploying changes

Canary

After changes have been fetched and otherwise git-wrangled on deployment.eqiad.wmnet, changes can be fetched down to mwdebug1002.eqiad.wmnet and tested via the X-Wikimedia-Debug header.

you@mwdebug1002:~$ scap pull

After changed have been fetched, ask patch-owner to test changes on mwdebug1002.eqiad.wmnet using X-Wikimedia-Debug.

Full deployment

After a change has been tested on mwdebug1002.eqiad.wmnet it can be deployed to all machines. To deploy the code you will run: scap sync-file <file> [message for SAL]. The code path passed to scap sync-file should be relative to /srv/mediawiki-staging.

The message you type after the file or directory name to be synced will appear in the Server Admin Log — wikitext is legal and can be useful. Copy/pasting the wikitext for that backport item from the Deployments calendar is easy. If the Gerrit change has an associated Phabricator task, mention the task ID in the message as appropriate. This will trigger Stashbot to reply back on tasks and indicate that the associated change was synced. You can use the backport-summary script (locally, not on the deployment host) to build the summary based on the Gerrit change URL.

you@deploy1002:/srv/mediawiki-staging$ scap sync-file [FILE|FOLDER] 'Backport: [[gerrit:[GERRIT-NUMBER]|[COMMIT-MESSAGE] ([PHABRICATOR-TASK])]]'

operations/mediawiki-config

Example:

you@deploy1002:/srv/mediawiki-staging$ scap sync-file wmf-config/InitialiseSettings.php 'Config: [[gerrit:444901|Enable FileExporter for sourceswiki (T198594)]]'

mediawiki/core

Example:

you@deploy1002:/srv/mediawiki-staging$ scap sync-file php-1.32.0-wmf.12/includes/cp/ChronologyProtector.php 'Backport: [[gerrit:445113|rdbms: fix value of ChronologyProtector::POSITION_COOKIE_TTL ([T194403])]]'

mediawiki/extensions and mediawiki/skins

Example:

you@deploy1002:/srv/mediawiki-staging$ scap sync-file php-1.32.0-wmf.12/extensions/WikimediaEvents 'Backport: [[gerrit:445377|Add event logging for WMDE fundraising banners (T197571)]]'

Purging

See also Multicast HTCP purging#One-off purge.

When a patch for mediawiki-config changes a file in /static, its public url must be purged from Varnish. For performance reasons, unversioned files in static have unconditional caching for up to a year. They rely on manual purges to propagate updates. This purge must be done with en.wikipedia.org as hostname of the url, regardless of which wiki the file relates to. This is because the cache for /static is shared between all wikis, and the canonical form internally for it is en.wikipedia.org.

• View url in browser. Example: https://en.wikipedia.org/static/images/project-logos/newikibooks.png
• Purge the url from mwmaint1002.eqiad.wmnet or wasat.codfw.wmnet:

you@mwmaint1002:~$ echo "https://en.wikipedia.org/static/images/project-logos/newikibooks.png" | mwscript purgeList.php

• Refresh url in browser.

Verification

After the sync and any purge is finished, monitor logs and ask patch-owner to, once again, test their changes to confirm the patch was deployed successfully. Make sure the patch-owner verifies it with X-Wikimedia-Debug turned off.

Reverting

If a patch doesn't work as expected, or causes errors, it will have to be reverted.

Process

Revert commit causing errors

you@deploy1002:[FOLDER]$ git revert [SHA1]

If the patch being reverted is a merge commit you will have to supply -m

you@deploy1002:[FOLDER]$ git revert [SHA1] -m1

Push code live BEFORE pushing patches to Gerrit

you@deploy1002:/srv/mediawiki-staging$ scap sync-file [FILE/FOLDER] 'Backport: Revert "[[gerrit:[NUMBER]|[TEXT] (T[NUMBER])]]"'

Push revert patch to Gerrit. Some setup might be needed. See Heterogeneous deployment/Train deploys#Setup.

you@deploy1002:[FOLDER]$ git push origin HEAD:refs/for/[BRANCH]/revert-[SHA1]

You will be prompted for your Gerrit https password.

Maintenance scripts

During the course of the window, you may encounter a patch that needs a maintenance script to be run as part of deployment. As noted earlier, maintenance scripts are run from mwmaint1002.eqiad.wmnet or mwmaint2001.codfw.wmnet.

For convenience, the most frequently run maintenance scripts are presented below.

tmux or screen

For long running scripts, it is recommended they are run in tmux or screen.

If you want to save terminal output, you can use script.

you@mwmaint1002:~$ script file_name
you@mwmaint1002:~$ # run command
...
you@mwmaint1002:~$ cat file_name
...

If you prefer tmux:

you@mwmaint1002:~$ tmux new -s 'backport'
...
you@mwmaint1002:~$ exit

If you need to leave in the middle you can do ctrl-b d to detach.

If you prefer screen:

you@mwmaint1002:~$ screen -D -RR backport
...
you@mwmaint1002:~$ exit

If you need to leave in the middle you can do ctrl-a d to detach.

Run a maintenance script on a group of wikis

See Wikimedia binaries#mwscriptwikiset.

Run a maintenance script on all wikis

See Wikimedia binaries#foreachwiki.

createExtensionTables

Allow to source SQL files to create tables for most extensions we deploy.

For example, to create on ar.wikipedia PageAssessments tables:

you@mwmaint1002:~ $ mwscript extensions/WikimediaMaintenance/createExtensionTables.php arwiki pageassessments

namespaceDupes

When a new namespace is added to an existing wiki, the namespaceDupes maintenance script should be run for that wiki. First do a dry run of namespaceDupes for the wiki (without --fix) as a sanity check. Then append --fix to fix namespace duplication:

you@mwmaint1002:~ $ mwscript namespaceDupes.php testwiki
you@mwmaint1002:~ $ mwscript namespaceDupes.php testwiki --fix

updateCollation

When the default collation is changed for a wiki, the updateCollation.php maintenance script will need to be run:

you@mwmaint1002:~$ mwscript updateCollation.php --wiki=iswiki --previous-collation=<VALUE>

Replace <VALUE> with what the wiki's previously configured collation name was in $wgCategoryCollation.

The default collation in MediaWiki is uppercase, as such in most cases when a wiki switches to a different collation the previous can be specified as --previous-collation=uppercase.

resetAuthenticationThrottle

See Increasing account creation threshold.