You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Backport windows/Deployers: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Lucas Werkmeister (WMDE)
m (remove SWAT from lead paragraph)
 
imported>Jeena Huneidi
(Add documentation for scap backport)
 
(20 intermediate revisions by 9 users not shown)
Line 1: Line 1:
This document is intended to provide detailed instructions for deployers as to how to run the backport windows. Hopefully, this document will prove useful to new deployers as well as provide a place for more experienced deployers to take notes on any tips and tricks they have discovered in the course of doing deployments.
{{Navigation MediaWiki deployment}}This document is intended to provide detailed instructions for deployers as to how to run the backport windows. Hopefully, this document will prove useful to new deployers as well as provide a place for more experienced deployers to take notes on any tips and tricks they have discovered in the course of doing deployments.


== General advice ==
== General advice ==


* Claim the SWAT window early to avoid confusion.
* Claim the window early to avoid confusion.
: When jouncebot pings deployers in '''{{irc|wikimedia-operations}}''', if you want to run that SWAT, say so <code>I can SWAT today!</code>
: When jouncebot pings deployers in '''{{irc|wikimedia-operations}}''', if you want to run that window, say so <code>I can do the deploys today!</code>
* Try to think out loud and be explicit.
* Try to think out loud and be explicit.
: If you are nervous about deploying a particular patch, mention it to the patch owner. It's better to have a conversation than to quietly fret over patches
: If you are nervous about deploying a particular patch, mention it to the patch owner. It's better to have a conversation than to quietly fret over patches.
* Be prepared.
* Be prepared.
: Open all your SSH connections and error logs before you start deploying code.
: Open the relevant SSH connections and browser tabs before you start deploying code.
* Release the SWAT window early when done to mark the rest of the window free.
* Release the window early when done to mark the rest of the window free.
: <code>!log EU SWAT done</code>, <code>!log Morning SWAT done</code>, <code>!log Evening SWAT done</code>, or something to that effect.
: <code>!log UTC morning deploys done</code>, <code>!log UTC evening deploys done</code>, <code>!log UTC late deploys done</code>, or something to that effect.


=== git ===
* Add git information to your terminal prompt. The <code>git-prompt</code> command is available on deployment servers. There are instructions for use in the [https://sources.debian.org/src/git/1:2.11.0-3+deb9u4/contrib/completion/git-prompt.sh/#L10-L31 comments at the beginning of the script]. One simple way to use it is to add the following to your <code>~/.profile</code>:<syntaxhighlight lang="bash">
 
* <code>git</code> can be confusing. It is more confusing when you're trying to accomplish something under pressure. There are a few config options that might be helpful.
 
# Use a git-aware prompt.<br>The git contrib <code>git-prompt</code> script is available on deployment servers. There are instructions for use in [https://sources.debian.org/src/git/1:2.11.0-3+deb9u4/contrib/completion/git-prompt.sh/#L10-L31 comments at the beginning of the file]. One simple way to use it is to add the following to your <code>~/.profile</code>: <syntaxhighlight lang=bash>
GIT_PS1_SHOWUNTRACKEDFILES=1
GIT_PS1_SHOWUNTRACKEDFILES=1
GIT_PS1_SHOWDIRTYSTATE=1
GIT_PS1_SHOWDIRTYSTATE=1
Line 23: Line 19:
PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '
PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '
</syntaxhighlight>
</syntaxhighlight>
# <code>status.submoduleSummary</code><br>Submodules have limited visibility in <code>git status</code> and it's easy to miss the <code>git submodule update</code> step. After adding <code>status.submoduleSummary</code> to your <code>~/.gitconfig</code> git will show you a short summary of submodule changes in <code>git status</code>. Set it by executing:<syntaxhighlight lang="shell-session">
* Enable Git configuration <code>status.submoduleSummary</code>. Submodules have limited visibility in <code>git status</code> and it's easy to miss the <code>git submodule update</code> step. Git can show you a short summary of submodule changes in <code>git status</code>. Enable it by executing:<syntaxhighlight lang="shell-session">
you@deploy1001:~$ git config --global status.submoduleSummary true
you@deploy1002:~$ git config --global status.submoduleSummary true
</syntaxhighlight>
</syntaxhighlight>
{{Note|The deployment of [[How_to_deploy_code#Security_patches|security patches]] for extensions does not entail a <code>git submodule update</code>, and so a given repository may appear "dirty".  This is the '''normal state''' of the repository.  See https://phabricator.wikimedia.org/T229285 for further related discussion.|reminder}}
{{Note|The deployment of [[How_to_deploy_code#Security_patches|security patches]] for extensions does not entail a <code>git submodule update</code>, and so a given repository may appear "dirty".  This is the '''normal state''' of the repository.  See https://phabricator.wikimedia.org/T229285 for further related discussion.|reminder}}


== SSH Connections and Error Logs ==
== SSH Connections and Error Logs: Set up before deploying ==


When running SWAT, it is helpful to watch error logs as you deploy so that you can be sure nothing you have just deployed is broken. Also, there are several machines on which you may need to run commands depending on the nature of the SWAT; it's good to open all SSH connections before you have to think about them.
When running the window, you'll want to see what's on the calendar, and watch error logs as you deploy so that you can be sure nothing you have just deployed is broken. Also, there are several machines on which you may need to run commands depending on the nature of the deploy; it's good to open all SSH connections before you have to think about them.


A [[SWAT deploys/Deployers/Script|script to set up these browser tabs and terminals automatically]] is available.
A [[Backport windows/Deployers/Script|script to set up these browser tabs and terminals automatically]] is available.


=== Browser tabs ===
=== Browser tabs ===


* The [[Deployments]] calendar
* [[Deployments|Deployments calendar]], links to patches that are scheduled for the window.
* [https://logstash.wikimedia.org/app/kibana#/dashboard/mediawiki-errors Logstash MediaWiki Errors]
* [https://versions.toolforge.org/ MediaWiki versions tool], to check what versions may need a backport.
* [https://logstash.wikimedia.org/app/kibana#/dashboard/mwdebug1002 Logstash Event log (mwdebug1002)]
* [https://integration.wikimedia.org/zuul/ Zuul Status dashboard], to watch the progress of CI for the patches being merged.
* [https://integration.wikimedia.org/zuul/ Zuul Dashboard]
* [https://logstash.wikimedia.org/app/dashboards#/view/mwdebug1002 Logstash: mwdebug], ensure no warnings or errors appear on the debug host when the patch owner does their verification.
* [https://tools.wmflabs.org/versions/ Wikimedia MediaWiki versions Dashboard]
* [https://logstash.wikimedia.org/app/dashboards#/view/mediawiki-errors Logstash: mediawiki-errors], ensure no new errors appear after patch is deployed to all servers.
* Also, ensure you're signed in on '''[https://gerrit.wikimedia.org/r/#/ Gerrit]


=== Terminal tabs ===
=== Terminal tabs ===


* [[mwlog1001|mwlog1001.eqiad.wmnet]] - to run <code>logspam-watch</code>, which you may wish to keep an eye during the SWAT window.
* [[mwlog1002|mwlog1002.eqiad.wmnet]] - to run <code>logspam-watch</code>, which you may wish to keep an eye during the window.
** You may also run <code>logspam</code> for a one-off listing of recent errors, suitable for grepping.
** You may also run <code>logspam</code> for a one-off listing of recent errors, suitable for grepping.
* [[deployment.eqiad.wmnet]] (which is an alias to a deployment host in the currently primary data centre) - This is where you stage changes.
* [[deployment.eqiad.wmnet]] (which is an alias to a deployment host in the currently primary data center) - This is where you stage changes.
* [[mwdebug1002]] - This is where you'll run <code>scap pull</code> to verify changes with [[X-Wikimedia-Debug#Staging_changes|X-Wikimedia-Debug]].
* [[mwdebug]] - This is where you'll run <code>scap pull</code> to verify changes with [[X-Wikimedia-Debug#Staging_changes|WikimediaDebug]].
* [[Mwmaint1002|mwmaint1002.eqiad.wmnet]] or [[Mwmaint2001|mwmaint2001.codfw.wmnet]]  - To run maintenance scripts (if needed).


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@laptop$ ssh mwlog1001.eqiad.wmnet
you@laptop$ ssh mwlog1002.eqiad.wmnet


you@mwlog1001:~$ logspam-watch
you@mwlog1002:~$ logspam-watch
</syntaxhighlight>
</syntaxhighlight>


Line 60: Line 54:
you@laptop$ ssh deployment.eqiad.wmnet
you@laptop$ ssh deployment.eqiad.wmnet


you@deploy1001:~$ cd /srv/mediawiki-staging
you@deploy1002:~$ cd /srv/mediawiki-staging
</syntaxhighlight>
</syntaxhighlight>


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@laptop$ ssh mwdebug1002.eqiad.wmnet
you@laptop$ ssh mwdebug.eqiad.wmnet # You may need to ssh to mwdebug1002 or mwdebug2002 depending on the active data center


you@mwdebug1002:~$ scap pull
you@mwdebug1002:~$ scap pull
Line 74: Line 68:
you@mwmaint1002:~$
you@mwmaint1002:~$
</syntaxhighlight>
</syntaxhighlight>
== Using [[scap#scap_backport|scap backport]] ==
<code>scap backport <change_number_or_url></code> can be used to backport both individual or multiple patches at once. It will handle merging (if needed) and deploying to testservers, wait for confirmation, and then run scap sync-world. For detailed instructions on how to manually deploy backports, continue reading.


== Merging and applying patches ==
== Merging and applying patches ==


For SWAT, you will be merging patches either for a wmf branch of [[phab:diffusion/MW/|mediawiki]] repositories (core, or a WMF-installed extension or skin), or the [[phab:diffusion/OMWC/|operations/mediawiki-config]] repo.
The [https://deploy-commands.toolforge.org/ deploy-commands tool], for any given gerrit change, produces a list of commands for deployment and revert that can be copy-pasted, and this tool is automatically linked to patches added to the [[Deployments|Deployment Calendar]] via [[Module:Gerrit]]. Don't copy-paste without reading carefully first, though; you should double-check the output in all cases.
 
You will be merging patches either for a wmf branch of [[phab:diffusion/MW/|mediawiki]] repositories (core, or a WMF-installed extension or skin), or the [[phab:diffusion/OMWC/|operations/mediawiki-config]] repo.
 
Check the [https://versions.toolforge.org/ MediaWiki versions tool] to confirm which branches may need a backport for a given patch.


=== Merging patches ===
=== Merging patches ===
Line 85: Line 86:
* to see how long (approximately) the test will take before it auto-merges.
* to see how long (approximately) the test will take before it auto-merges.


It's a good practice to put <code>SWAT</code> as the comment when you <code>+2</code> before you click <code>Publish Comments</code> to ensure that there is a record of why you approved the patch.
It's a good practice to put <code>Backport</code> as the comment when you <code>+2</code> before you click <code>Publish Comments</code> to ensure that there is a record of why you approved the patch.


=== Fetching patches ===
=== Fetching patches ===
Line 98: Line 99:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@deploy1001:/srv/mediawiki-staging$ git status
you@deploy1002:/srv/mediawiki-staging$ git status


you@deploy1001:/srv/mediawiki-staging$ git fetch
you@deploy1002:/srv/mediawiki-staging$ git fetch


you@deploy1001:/srv/mediawiki-staging$ git log -p HEAD..@{u}
you@deploy1002:/srv/mediawiki-staging$ git log -p HEAD..@{u}


you@deploy1001:/srv/mediawiki-staging$ git rebase
you@deploy1002:/srv/mediawiki-staging$ git rebase
</syntaxhighlight>
</syntaxhighlight>


Line 110: Line 111:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git status
you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git status


you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git fetch
you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git fetch


you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git log HEAD..@{u}
you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git log -p HEAD..@{u}


you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git rebase
you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git rebase
</syntaxhighlight>
</syntaxhighlight>


Line 123: Line 124:
As soon as the change to an extension or skin gets merged, Gerrit bumps the associated submodule in the <code>wmf/*</code> branch of <code>mediawiki/core</code>. To deploy, you thus just have to fetch that parent repository, verify the difference between the current state (<code>HEAD</code>) and the tracked remote branch (<code>@{u}</code>), rebase and update the submodule:
As soon as the change to an extension or skin gets merged, Gerrit bumps the associated submodule in the <code>wmf/*</code> branch of <code>mediawiki/core</code>. To deploy, you thus just have to fetch that parent repository, verify the difference between the current state (<code>HEAD</code>) and the tracked remote branch (<code>@{u}</code>), rebase and update the submodule:
<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git status
you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git status


you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git fetch
you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git fetch


you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git log -p HEAD..@{u}
you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git log -p HEAD..@{u}


you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git rebase
you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git rebase


you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git status [extensions|skins]/[NAME]
you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git status [extensions|skins]/[NAME]


you@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git submodule update [extensions|skins]/[NAME]
you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git submodule update [extensions|skins]/[NAME]
</syntaxhighlight>
</syntaxhighlight>


Line 159: Line 160:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@deploy1001:/srv/mediawiki-staging$ scap sync-file [FILE|FOLDER] 'Backport: [[gerrit:[GERRIT-NUMBER]|[COMMIT-MESSAGE] ([PHABRICATOR-TASK])]]'
you@deploy1002:/srv/mediawiki-staging$ scap sync-file [FILE|FOLDER] 'Backport: [[gerrit:[GERRIT-NUMBER]|[COMMIT-MESSAGE] ([PHABRICATOR-TASK])]]'
</syntaxhighlight>
</syntaxhighlight>


Line 167: Line 168:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@deploy1001:/srv/mediawiki-staging$ scap sync-file wmf-config/InitialiseSettings.php 'Config: [[gerrit:444901|Enable FileExporter for sourceswiki (T198594)]]'
you@deploy1002:/srv/mediawiki-staging$ scap sync-file wmf-config/InitialiseSettings.php 'Config: [[gerrit:444901|Enable FileExporter for sourceswiki (T198594)]]'
</syntaxhighlight>
</syntaxhighlight>


Line 175: Line 176:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@deploy1001:/srv/mediawiki-staging$ scap sync-file php-1.32.0-wmf.12/includes/cp/ChronologyProtector.php 'Backport: [[gerrit:445113|rdbms: fix value of ChronologyProtector::POSITION_COOKIE_TTL ([T194403])]]'
you@deploy1002:/srv/mediawiki-staging$ scap sync-file php-1.32.0-wmf.12/includes/cp/ChronologyProtector.php 'Backport: [[gerrit:445113|rdbms: fix value of ChronologyProtector::POSITION_COOKIE_TTL ([T194403])]]'
</syntaxhighlight>
</syntaxhighlight>


Line 183: Line 184:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@deploy1001:/srv/mediawiki-staging$ scap sync-file php-1.32.0-wmf.12/extensions/WikimediaEvents 'Backport: [[gerrit:445377|Add event logging for WMDE fundraising banners (T197571)]]'
you@deploy1002:/srv/mediawiki-staging$ scap sync-file php-1.32.0-wmf.12/extensions/WikimediaEvents 'Backport: [[gerrit:445377|Add event logging for WMDE fundraising banners (T197571)]]'
</syntaxhighlight>
</syntaxhighlight>


Line 192: Line 193:


* View url in browser. Example: https://en.wikipedia.org/static/images/project-logos/newikibooks.png
* View url in browser. Example: https://en.wikipedia.org/static/images/project-logos/newikibooks.png
* Purge the url from <code>mwmaint1002.eqiad.wmnet</code> or <code>wasat.codfw.wmnet</code>:
* Purge the url from a [[Maintenance server]]:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@mwmaint1002:~$ echo "https://en.wikipedia.org/static/images/project-logos/newikibooks.png" | mwscript purgeList.php
you@mwmaint1002:~$ echo 'https://en.wikipedia.org/static/images/project-logos/newikibooks.png' | mwscript purgeList.php
</syntaxhighlight>
</syntaxhighlight>


Line 210: Line 211:


=== Process ===
=== Process ===
Use <code>scap backport --revert <change_number_or_url></code> to automatically revert and deploy code, or follow the instructions below. Note that the revert command will wait for the patches to merge before deploying, so in an emergency it may be ideal to revert manually.


Revert commit causing errors
Revert commit causing errors


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@deploy1001:[FOLDER]$ git revert [SHA1]
you@deploy1002:[FOLDER]$ git revert [SHA1]
</syntaxhighlight>
</syntaxhighlight>


Line 220: Line 223:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@deploy1001:[FOLDER]$ git revert [SHA1] -m1
you@deploy1002:[FOLDER]$ git revert [SHA1] -m1
</syntaxhighlight>
</syntaxhighlight>


Line 226: Line 229:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@deploy1001:/srv/mediawiki-staging$ scap sync-file [FILE/FOLDER] 'SWAT: Revert "[[gerrit:[NUMBER]|[TEXT] (T[NUMBER])]]"'
you@deploy1002:/srv/mediawiki-staging$ scap sync-file [FILE/FOLDER] 'Backport: Revert "[[gerrit:[NUMBER]|[TEXT] (T[NUMBER])]]"'
</syntaxhighlight>
</syntaxhighlight>


Line 232: Line 235:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@deploy1001:[FOLDER]$ git push origin HEAD:refs/for/[BRANCH]/revert-[SHA1]
you@deploy1002:[FOLDER]$ git push origin HEAD:refs/for/[BRANCH]%topic=revert-[SHA1]
</syntaxhighlight>
</syntaxhighlight>


Line 239: Line 242:
== Maintenance scripts ==
== Maintenance scripts ==


During the course of SWAT, you may encounter a patch that needs a maintenance script to be run as part of deployment. As noted earlier, maintenance scripts are run from <code>mwmaint1002.eqiad.wmnet</code> or <code>mwmaint2001.codfw.wmnet</code>.
During the course of the window, you may encounter a patch that needs a maintenance script to be run as part of deployment. As noted earlier, maintenance scripts are run from <code>mwmaint1002.eqiad.wmnet</code> or <code>mwmaint2001.codfw.wmnet</code>.


For convenience, the most frequently run maintenance scripts are presented below.
For convenience, the most frequently run maintenance scripts are presented below.
Line 260: Line 263:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@mwmaint1002:~$ tmux new -s 'swat'
you@mwmaint1002:~$ tmux new -s 'backport'
...
...
you@mwmaint1002:~$ exit
you@mwmaint1002:~$ exit
Line 270: Line 273:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
you@mwmaint1002:~$ screen -D -RR swat
you@mwmaint1002:~$ screen -D -RR backport
...
...
you@mwmaint1002:~$ exit
you@mwmaint1002:~$ exit
Line 318: Line 321:


[[Category:How-To]]
[[Category:How-To]]
[[Category:Deployment]]

Latest revision as of 22:44, 25 August 2022

Deployments

This document is intended to provide detailed instructions for deployers as to how to run the backport windows. Hopefully, this document will prove useful to new deployers as well as provide a place for more experienced deployers to take notes on any tips and tricks they have discovered in the course of doing deployments.

General advice

  • Claim the window early to avoid confusion.
When jouncebot pings deployers in #wikimedia-operations connect, if you want to run that window, say so I can do the deploys today!
  • Try to think out loud and be explicit.
If you are nervous about deploying a particular patch, mention it to the patch owner. It's better to have a conversation than to quietly fret over patches.
  • Be prepared.
Open the relevant SSH connections and browser tabs before you start deploying code.
  • Release the window early when done to mark the rest of the window free.
!log UTC morning deploys done, !log UTC evening deploys done, !log UTC late deploys done, or something to that effect.
  • Add git information to your terminal prompt. The git-prompt command is available on deployment servers. There are instructions for use in the comments at the beginning of the script. One simple way to use it is to add the following to your ~/.profile:
    GIT_PS1_SHOWUNTRACKEDFILES=1
    GIT_PS1_SHOWDIRTYSTATE=1
    GIT_PS1_SHOWUPSTREAM="auto verbose"
    . /etc/bash_completion.d/git-prompt
    PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '
    
  • Enable Git configuration status.submoduleSummary. Submodules have limited visibility in git status and it's easy to miss the git submodule update step. Git can show you a short summary of submodule changes in git status. Enable it by executing:
    you@deploy1002:~$ git config --global status.submoduleSummary true
    

SSH Connections and Error Logs: Set up before deploying

When running the window, you'll want to see what's on the calendar, and watch error logs as you deploy so that you can be sure nothing you have just deployed is broken. Also, there are several machines on which you may need to run commands depending on the nature of the deploy; it's good to open all SSH connections before you have to think about them.

A script to set up these browser tabs and terminals automatically is available.

Browser tabs

Terminal tabs

  • mwlog1002.eqiad.wmnet - to run logspam-watch, which you may wish to keep an eye during the window.
    • You may also run logspam for a one-off listing of recent errors, suitable for grepping.
  • deployment.eqiad.wmnet (which is an alias to a deployment host in the currently primary data center) - This is where you stage changes.
  • mwdebug - This is where you'll run scap pull to verify changes with WikimediaDebug.
you@laptop$ ssh mwlog1002.eqiad.wmnet

you@mwlog1002:~$ logspam-watch
you@laptop$ ssh deployment.eqiad.wmnet

you@deploy1002:~$ cd /srv/mediawiki-staging
you@laptop$ ssh mwdebug.eqiad.wmnet # You may need to ssh to mwdebug1002 or mwdebug2002 depending on the active data center

you@mwdebug1002:~$ scap pull
you@laptop$ ssh mwmaint1002.eqiad.wmnet

you@mwmaint1002:~$

Using scap backport

scap backport <change_number_or_url> can be used to backport both individual or multiple patches at once. It will handle merging (if needed) and deploying to testservers, wait for confirmation, and then run scap sync-world. For detailed instructions on how to manually deploy backports, continue reading.

Merging and applying patches

The deploy-commands tool, for any given gerrit change, produces a list of commands for deployment and revert that can be copy-pasted, and this tool is automatically linked to patches added to the Deployment Calendar via Module:Gerrit. Don't copy-paste without reading carefully first, though; you should double-check the output in all cases.

You will be merging patches either for a wmf branch of mediawiki repositories (core, or a WMF-installed extension or skin), or the operations/mediawiki-config repo.

Check the MediaWiki versions tool to confirm which branches may need a backport for a given patch.

Merging patches

When +2ing patches, it's often helpful to have the Zuul Dashboard open

  • to ensure that Zuul is picking up your changes,
  • to see how long (approximately) the test will take before it auto-merges.

It's a good practice to put Backport as the comment when you +2 before you click Publish Comments to ensure that there is a record of why you approved the patch.

Fetching patches

After the patch has merged in Gerrit, you need to fetch it down to deployment.eqiad.wmnet. Make sure that the code you fetch down to deployment.eqiad.wmnet is the code you expected to fetch down.

Use git log -p HEAD..@{u} after you git fetch to check that the patch(es) you fetched down were the same ones the you +2'd. If they aren't, poke the person that wrote the patch in #wikimedia-operations connect to figure out what to do with the fetched code. It's always better to ask than to do something silently and unilaterally.

Process is slightly different for operations/mediawiki-config, mediawiki/core, mediawiki/extensions and mediawiki/skins.

operations/mediawiki-config

you@deploy1002:/srv/mediawiki-staging$ git status

you@deploy1002:/srv/mediawiki-staging$ git fetch

you@deploy1002:/srv/mediawiki-staging$ git log -p HEAD..@{u}

you@deploy1002:/srv/mediawiki-staging$ git rebase

mediawiki/core

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git status

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git fetch

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git log -p HEAD..@{u}

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git rebase

mediawiki/extensions and mediawiki/skins

As soon as the change to an extension or skin gets merged, Gerrit bumps the associated submodule in the wmf/* branch of mediawiki/core. To deploy, you thus just have to fetch that parent repository, verify the difference between the current state (HEAD) and the tracked remote branch (@{u}), rebase and update the submodule:

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git status

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git fetch

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git log -p HEAD..@{u}

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git rebase

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git status [extensions|skins]/[NAME]

you@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git submodule update [extensions|skins]/[NAME]
Security Patches

Refer to How to deploy code -> Security patches for information about security patches.

Deploying changes

Canary

After changes have been fetched and otherwise git-wrangled on deployment.eqiad.wmnet, changes can be fetched down to mwdebug1002.eqiad.wmnet and tested via the X-Wikimedia-Debug header.

you@mwdebug1002:~$ scap pull

After changed have been fetched, ask patch-owner to test changes on mwdebug1002.eqiad.wmnet using X-Wikimedia-Debug.

Full deployment

After a change has been tested on mwdebug1002.eqiad.wmnet it can be deployed to all machines. To deploy the code you will run: scap sync-file <file> [message for SAL]. The code path passed to scap sync-file should be relative to /srv/mediawiki-staging.

The message you type after the file or directory name to be synced will appear in the Server Admin Log — wikitext is legal and can be useful. Copy/pasting the wikitext for that backport item from the Deployments calendar is easy. If the Gerrit change has an associated Phabricator task, mention the task ID in the message as appropriate. This will trigger Stashbot to reply back on tasks and indicate that the associated change was synced. You can use the backport-summary script (locally, not on the deployment host) to build the summary based on the Gerrit change URL.

you@deploy1002:/srv/mediawiki-staging$ scap sync-file [FILE|FOLDER] 'Backport: [[gerrit:[GERRIT-NUMBER]|[COMMIT-MESSAGE] ([PHABRICATOR-TASK])]]'

operations/mediawiki-config

Example:

you@deploy1002:/srv/mediawiki-staging$ scap sync-file wmf-config/InitialiseSettings.php 'Config: [[gerrit:444901|Enable FileExporter for sourceswiki (T198594)]]'

mediawiki/core

Example:

you@deploy1002:/srv/mediawiki-staging$ scap sync-file php-1.32.0-wmf.12/includes/cp/ChronologyProtector.php 'Backport: [[gerrit:445113|rdbms: fix value of ChronologyProtector::POSITION_COOKIE_TTL ([T194403])]]'

mediawiki/extensions and mediawiki/skins

Example:

you@deploy1002:/srv/mediawiki-staging$ scap sync-file php-1.32.0-wmf.12/extensions/WikimediaEvents 'Backport: [[gerrit:445377|Add event logging for WMDE fundraising banners (T197571)]]'

Purging

See also Multicast HTCP purging#One-off purge.

When a patch for mediawiki-config changes a file in /static, its public url must be purged from Varnish. For performance reasons, unversioned files in static have unconditional caching for up to a year. They rely on manual purges to propagate updates. This purge must be done with en.wikipedia.org as hostname of the url, regardless of which wiki the file relates to. This is because the cache for /static is shared between all wikis, and the canonical form internally for it is en.wikipedia.org.

you@mwmaint1002:~$ echo 'https://en.wikipedia.org/static/images/project-logos/newikibooks.png' | mwscript purgeList.php
  • Refresh url in browser.

Verification

After the sync and any purge is finished, monitor logs and ask patch-owner to, once again, test their changes to confirm the patch was deployed successfully. Make sure the patch-owner verifies it with X-Wikimedia-Debug turned off.

Reverting

If a patch doesn't work as expected, or causes errors, it will have to be reverted.

Process

Use scap backport --revert <change_number_or_url> to automatically revert and deploy code, or follow the instructions below. Note that the revert command will wait for the patches to merge before deploying, so in an emergency it may be ideal to revert manually.

Revert commit causing errors

you@deploy1002:[FOLDER]$ git revert [SHA1]

If the patch being reverted is a merge commit you will have to supply -m

you@deploy1002:[FOLDER]$ git revert [SHA1] -m1

Push code live BEFORE pushing patches to Gerrit

you@deploy1002:/srv/mediawiki-staging$ scap sync-file [FILE/FOLDER] 'Backport: Revert "[[gerrit:[NUMBER]|[TEXT] (T[NUMBER])]]"'

Push revert patch to Gerrit. Some setup might be needed. See Heterogeneous deployment/Train deploys#Setup.

you@deploy1002:[FOLDER]$ git push origin HEAD:refs/for/[BRANCH]%topic=revert-[SHA1]

You will be prompted for your Gerrit https password.

Maintenance scripts

During the course of the window, you may encounter a patch that needs a maintenance script to be run as part of deployment. As noted earlier, maintenance scripts are run from mwmaint1002.eqiad.wmnet or mwmaint2001.codfw.wmnet.

For convenience, the most frequently run maintenance scripts are presented below.

tmux or screen

For long running scripts, it is recommended they are run in tmux or screen.

If you want to save terminal output, you can use script.

you@mwmaint1002:~$ script file_name
you@mwmaint1002:~$ # run command
...
you@mwmaint1002:~$ cat file_name
...

If you prefer tmux:

you@mwmaint1002:~$ tmux new -s 'backport'
...
you@mwmaint1002:~$ exit

If you need to leave in the middle you can do ctrl-b d to detach.

If you prefer screen:

you@mwmaint1002:~$ screen -D -RR backport
...
you@mwmaint1002:~$ exit

If you need to leave in the middle you can do ctrl-a d to detach.

Run a maintenance script on a group of wikis

See Wikimedia binaries#mwscriptwikiset.

Run a maintenance script on all wikis

See Wikimedia binaries#foreachwiki.

createExtensionTables

Allow to source SQL files to create tables for most extensions we deploy.

For example, to create on ar.wikipedia PageAssessments tables:

you@mwmaint1002:~ $ mwscript extensions/WikimediaMaintenance/createExtensionTables.php arwiki pageassessments

namespaceDupes

When a new namespace is added to an existing wiki, the namespaceDupes maintenance script should be run for that wiki. First do a dry run of namespaceDupes for the wiki (without --fix) as a sanity check. Then append --fix to fix namespace duplication:

you@mwmaint1002:~ $ mwscript namespaceDupes.php testwiki
you@mwmaint1002:~ $ mwscript namespaceDupes.php testwiki --fix

updateCollation

When the default collation is changed for a wiki, the updateCollation.php maintenance script will need to be run:

you@mwmaint1002:~$ mwscript updateCollation.php --wiki=iswiki --previous-collation=<VALUE>

Replace <VALUE> with what the wiki's previously configured collation name was in $wgCategoryCollation.

The default collation in MediaWiki is uppercase, as such in most cases when a wiki switches to a different collation the previous can be specified as --previous-collation=uppercase.

resetAuthenticationThrottle

See Increasing account creation threshold.