You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Heterogeneous deployment/Train deploys: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Jforrester
imported>Krinkle
No edit summary
 
(62 intermediate revisions by 15 users not shown)
Line 1: Line 1:
[[File:MTR_CSR_Sifang_EMU_in_Shek_Kong_Stabling_Sidings_201710.jpg|thumb|500x500px|Bring new code in a fast, safe and efficient way!]]
{{Navigation MediaWiki deployment}}[[File:Trainbows_Not_Painbows1.svg|frameless|none|500px]]
{{Navigation MediaWiki deployment}}
<br>__TOC__
== Pairing on the Train ==
 
== Pairing on the Train ==


As of October 2019, there are two people assigned to each week's train:  One as primary, and one as backup.  These are rough guidelines for sharing the work, and should be improved as we learn more.
As of October 2019, there are two people assigned to each week's train:  One as primary, and one as backup.  These are rough guidelines for sharing the work, and should be improved as we learn more.
Line 13: Line 14:
* If the train is blocked or there are any other issues, communicate the transfer of responsibility on the train blocker ticket by assigning it to the responsible party and leaving a note.
* If the train is blocked or there are any other issues, communicate the transfer of responsibility on the train blocker ticket by assigning it to the responsible party and leaving a note.


== Breakage ==
== Initial setup==
 
SSH into '''deploy1002.eqiad.wmnet''' and run
<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:~$ git clone https://gerrit.wikimedia.org/r/mediawiki/tools/release
</syntaxhighlight>
 
==Breakage==


There will be times when this process does not go smoothly. There are [[Deployments/Holding_the_train|guidelines]] for what do to when that happens.
There will be times when this process does not go smoothly. There are [[Deployments/Holding_the_train|guidelines]] for what do to when that happens.
Line 19: Line 27:
In general, '''if there is an unexplained error that occurs within 1 hour of a train deployment — always roll back the train'''. Rolling back the train to eliminate it as the cause of unexplained breakage can be especially important if there are many ongoing possible causes for issues as this helps to eliminate one of those causes as the source of problems.
In general, '''if there is an unexplained error that occurs within 1 hour of a train deployment — always roll back the train'''. Rolling back the train to eliminate it as the cause of unexplained breakage can be especially important if there are many ongoing possible causes for issues as this helps to eliminate one of those causes as the source of problems.


=== Rollback ===
===Rollback===


To rollback a wikiversion change, it should be pretty quick. Go ahead and rollback production before you send patches up to gerrit since waiting on Jenkins may take a while:
To rollback a wikiversion change, it should be pretty quick. Go ahead and rollback production before you send patches up to gerrit since waiting on Jenkins may take a while:
Line 26: Line 34:
USERNAME@deploy1001:/srv/mediawiki-staging$ git revert $(git log -1 --format=%H -- wikiversions.json)
USERNAME@deploy1001:/srv/mediawiki-staging$ git revert $(git log -1 --format=%H -- wikiversions.json)
USERNAME@deploy1001:/srv/mediawiki-staging$ scap sync-wikiversions 'Revert "group[0|1] wikis to [VERSION]"'
USERNAME@deploy1001:/srv/mediawiki-staging$ scap sync-wikiversions 'Revert "group[0|1] wikis to [VERSION]"'
USERNAME@deploy1001:/srv/mediawiki-staging$ # Now that you've synced the revert, push patches up to gerrit, you have to run git commit --amend to get the changeid
 
# Now that you've synced the revert, push patches up to gerrit, you have to run git commit --amend to get the changeid
# Ideally, you should also add the train blocker task id to the Bug: field for this commit
USERNAME@deploy1001:/srv/mediawiki-staging$ git commit --amend
USERNAME@deploy1001:/srv/mediawiki-staging$ git commit --amend
USERNAME@deploy1001:/srv/mediawiki-staging$ git push origin HEAD:refs/for/master/[VERSION]%l=Code-Review+2
USERNAME@deploy1001:/srv/mediawiki-staging$ git push origin HEAD:refs/for/master%topic=[VERSION],l=Code-Review+2
</syntaxhighlight>
</syntaxhighlight>


Line 34: Line 44:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging$ git push origin HEAD:refs/for/master/1.34.0-wmf.0%l=Code-Review+2
USERNAME@deploy1001:/srv/mediawiki-staging$ git push origin HEAD:refs/for/master%topic=1.34.0-wmf.0,l=Code-Review+2
</syntaxhighlight>
</syntaxhighlight>


Line 41: Line 51:
*[[#Update roadmap]].
*[[#Update roadmap]].


=== Places to Watch for Breakage ===
===Places to Watch for Breakage===


Train deployers should check for breakage as they are rolling out train as they are effectively the first line of defense for train deploys. Some of the places to watch for breakage:
Train deployers should check for breakage as they are rolling out the train as they are effectively the first line of defense for train deploys. Some of the places to watch for breakage:


* IRC
*IRC
** primary channel is {{irc|wikimedia-operations}}
**Primary channel is {{irc|wikimedia-operations}}
** useful channels are {{irc|mediawiki-core}} {{irc|wikimedia-dev}}
**Useful channels are {{irc|mediawiki-core}} {{irc|wikimedia-dev}}
** for more channels see [https://www.mediawiki.org/wiki/MediaWiki_on_IRC MediaWiki on IRC] and [https://meta.wikimedia.org/wiki/IRC/Channels IRC/Channels]
**For more channels see [[mw:MediaWiki_on_IRC|MediaWiki on IRC]] and [[metawiki:IRC/Channels|IRC/Channels]]
* [[Mwlog1001|mwlog1001]]
*Logs
** [[Wikimedia_binaries#logspam-watch|logspam-watch]]
**[[mwlog1001]]:
** Logfiles in <code>/srv/mw-log</code>
***[[Wikimedia_binaries#logspam-watch|logspam-watch]]
* [https://logstash.wikimedia.org/app/kibana#/dashboard/Fatal-Monitor Logstash Fatal Monitor]
*** logfiles in <code>/srv/mw-log</code>
*[https://logstash.wikimedia.org/app/kibana#/dashboard/mediawiki-errors Logstash MediaWiki Errors]
**Logstash [https://logstash.wikimedia.org/app/dashboards#/view/mediawiki-errors mediawiki-errors] dashboard
*Logstash "mediawiki-new-errors" dashboard (linked from logstash front page)
**Logstash "mediawiki-new-errors" dashboard (linked from logstash front page)
**[https://logstash.wikimedia.org/app/kibana#/dashboard/dfcf7b70-1aaa-11e9-b4bc-db12fe15ab31 Showing only timeout errors] (see T204871)
**See the [[phab:tag/wikimedia-production-error/|Wikimedia-production-error workboard]] for known issues
* Group-specific Logstash Dashboards:
**Logstash [https://logstash.wikimedia.org/app/dashboards#/view/AXDBY8Qhh3Uj6x1zCF56 mw-client-errors] dashboard
** [https://logstash.wikimedia.org/app/kibana#/dashboard/group0 group0]
***New errors appearing more than 1000 times in a 12 hour period should be considered blockers
** [https://logstash.wikimedia.org/app/kibana#/dashboard/group1 group1]
***See also [https://grafana.wikimedia.org/d/000000566/overview?viewPanel=16&orgId=1 Grafana dashboard] with summary of average error rate over time
* [https://grafana.wikimedia.org/dashboard/db/varnish-http-errors?refresh=5m&orgId=1 Grafana Varnish error-rate dashboard] (HTTP 5XX % should have 3+ 0s after the decimal point, e.g. 0.0001%)
*[https://grafana.wikimedia.org/ Grafana]
* [https://grafana.wikimedia.org/d/000000612/frontend-responses-nginx-vs-varnish?orgId=1&from=now-15m&to=now Grafana Frontend Responses NGINX vs Varnish]
**[https://grafana.wikimedia.org/d/000000503/varnish-http-errors?refresh=5m&orgId=1 Varnish http-errors dashboard] (HTTP 5XX % should have 3+ 0s after the decimal point, e.g. 0.0001%)
* [https://grafana.wikimedia.org/d/000000102/production-logging Grafana Production Logging]
**[https://grafana.wikimedia.org/d/000000612/frontend-responses-nginx-vs-varnish?orgId=1&from=now-15m&to=now Frontend Responses NGINX vs Varnish]
* [https://grafana.wikimedia.org/d/000000566/overview?panelId=15&fullscreen&orgId=1&from=now-7d&to=now Minerva Client Errors] - Browser JS errors count (only wikipedias on mobile)
**[https://grafana.wikimedia.org/d/000000102/production-logging Production Logging]
**[https://grafana.wikimedia.org/d/000000566/overview?panelId=15&fullscreen&orgId=1&from=now-7d&to=now Minerva Client Errors] - Browser JS errors count (only wikipedias on mobile)
**[https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?orgId=1&from=now-1h&to=now&refresh=30s Application Servers RED Dashboard]


=== If the train is blocked===
===If the train is blocked===


*A task will be assigned to you, for example [https://phabricator.wikimedia.org/T191059 T191059] (1.32.0-wmf.13 deployment blockers)
*A task will be assigned to you, for example [[phab:T191059|T191059]] (1.32.0-wmf.13 deployment blockers) (you can see that week's task at https://train-blockers.toolforge.org)
*Any open subtasks block the train from moving forward. This means no further deployments until the blockers are resolved.
*Any open subtasks block the train from moving forward. This means no further deployments until the blockers are resolved.


Line 73: Line 85:
If there are blocking tasks, please do the following:
If there are blocking tasks, please do the following:


*Make sure all tasks blocking train are set to <code>UBN!</code> priority in phabricator
* Make sure all tasks blocking train are set to <code>UBN!</code> priority in phabricator
*Comment on the task asking for an ETA or if this can be solved by reverting a recent commit.
*Comment on the task asking for an ETA or if this can be solved by reverting a recent commit.
*Send e-mail to:
*Send e-mail to:
**[https://lists.wikimedia.org/mailman/listinfo/ops ops@lists.wikimedia.org]
**[[mail:ops|ops@lists.wikimedia.org]]
**[https://lists.wikimedia.org/mailman/listinfo/wikitech-l wikitech-l@lists.wikimedia.org]
**[[mail:wikitech-l|wikitech-l@lists.wikimedia.org]]
** Ping private [https://app.slack.com/client/T024KLHS4/C01R06P8D1B/ #engineering-all Slack channel]
**Subject: <code>[Train] {version} status update</code>
**Subject: <code>[Train] {version} status update</code>
**Body<syntaxhighlight lang="text">The {version} version of MediaWiki is blocked[0].
**Body<syntaxhighlight lang="text">The {version} version of MediaWiki is blocked[0].
Line 94: Line 107:


[0]. <{link to phab task for train}>
[0]. <{link to phab task for train}>
[1]. <https://tools.wmflabs.org/versions/></syntaxhighlight>
[1]. <https://versions.toolforge.org/></syntaxhighlight>
*Add relevant people (see [https://www.mediawiki.org/wiki/Developers/Maintainers Developers/Maintainers]) to the blocking task
*Add relevant people (see [[mw:Developers/Maintainers|Developers/Maintainers]]) to the blocking task
* Ping relevant people in IRC
*Ping relevant people in IRC
* Once train is unblocked be sure to thank the folks who helped unblock it
*Once train is unblocked be sure to thank the folks who helped unblock it


==Monday: Sync up with your deployment partner==
==Weekly steps==
 
=== Monday: Sync up with your deployment partner===


See the [[#Pairing on the Train|train pairing]] section above.
See the [[#Pairing on the Train|train pairing]] section above.


==Tuesday: New branch creation and deploy==
===Tuesday: New branch creation and deploy===
====Before the deploy window====


The new branch can be created in Gerrit from anywhere. It is often faster to do this step on a host in the cluster to minimize the time needed to clone from Gerrit.
Depending on how practiced you are and where you choose to run commands (full clones of mediawiki-core from outside the cluster can take a while), the steps will typically take 45 to 90 minutes.


===Before the deploy window===
; Short-form instructions
{| class="wikitable"
! colspan="2" |Step
!host
! command
!example
|-
| P-0
|'''Verify branch cut job worked'''
|Your laptop
| colspan="2" |The branch cut is performed by a Jenkins job that runs on Tuesdays at 02:00 UTC on the [https://releases-jenkins.wikimedia.org/ releases-jenkins] instance.  The job creates and merges the wmf/* branch.  It also builds and posts the changelog.


Depending on how practiced you are and where you choose to run commands (full clones of mediawiki-core from outside the cluster can take a while) the steps will typically take 45 to 90 minutes.
Navigate to [[gerrit:#/q/owner:%2522TrainBranchBot+%253Cmmodell%252Btrainbranchbot%2540wikimedia.org%253E%2522+%2522Branch+commit%2522|Gerrit]] to find the branch commit that the job created.  If there is no commit for the current train branch shown in Gerrit, you can troubleshoot via the [https://releases-jenkins.wikimedia.org/job/Automatic%20branch%20cut/ releases-jenkins job].
|-
|P-3
|'''Enter screen (or tmux if you prefer)'''


====Setup====


The script is run as your regular user member of the <code>wikidev</code> group (as of Feb 16th 2016).
Note<ref>If you need to leave in the middle you can do <code>ctrl-a d</code> to detach and <code>screen -r train</code> to attach.</ref>
|'''deploy1002.eqiad.wmnet'''
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:~$ screen -D -RR train
</syntaxhighlight>
|-
|P-4
|'''Set local ssh-agent in session'''
|deploy1002
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:~$ eval $(ssh-agent)
USERNAME@deploy1002:~$ ssh-add .ssh/id_ed25519
</syntaxhighlight>
|-
|P-5
|'''Clone new branch in production, apply patches, deploy to test-wikis, and cleanup old versions'''
🐌 Note: this step takes > 1 hour.
|deploy1002
|<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:~$ scap stage-train [VERSION]
</syntaxhighlight>
|<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:~$ scap stage-train 1.34.0-wmf.0
</syntaxhighlight>
You can also run this command with the <code>--dry-run</code> option to see all the commands it will try to run:
<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:~$ scap stage-train --dry-run 1.34.0-wmf.0
</syntaxhighlight>
|-
! colspan="5" |'''Wait for the deploy window'''
|}


Configure Git. You will need to ensure your <code>user.name</code> and <code>user.email</code> is set correctly. You will also need to ensure that you push to gerrit via <code>ssh</code> (as of Mar 18th, 2019).
;Full instructions
<div class="mw-collapsible mw-collapsed" style="margin:1em; border: 1px dashed grey; padding: 2em 1em;">
=====Setup=====


<syntaxhighlight lang="shell-session">
The script to cut a branch is run on your local machine (as of Jan 2020).
USERNAME@MACIHNE-NAME:~$ ssh deployment.eqiad.wmnet


USERNAME@deploy1001:~$ git config --global user.name "[FIRST-NAME] [LAST-NAME]"
'''Local <code>.netrc</code> setup'''
USERNAME@deploy1001:~$ git config --global user.email "[USERNAME]@[DOMAIN]"
USERNAME@deploy1001:~$ git config --global url.ssh://[GERRIT-USERNAME]@gerrit.wikimedia.org:29418.pushInsteadOf https://gerrit.wikimedia.org/r
</syntaxhighlight>


Create a new ssh key to use from the deployment server.
Create a .netrc file in your home directory with the following content.


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:~$ ssh-keygen -t ed25519
you@yourlaptop:~$ vim .netrc
Enter file in which to save the key (/home/USERNAME/.ssh/id_ed25519): <enter>
machine gerrit.wikimedia.org login [USERNAME] password [PASSWORD]
Enter passphrase (empty for no passphrase): <passphrase>
Enter same passphrase again: <passphrase>
Your identification has been saved in /home/USERNAME/.ssh/id_ed25519.
Your public key has been saved in /home/USERNAME/.ssh/id_ed25519.pub.
</syntaxhighlight>
</syntaxhighlight>


Add your new <code>~/.ssh/id_ed25519.pub</code> key to Gerrit:
Username and password can obtained from Gerrit:
 
*In the new UI go to [[gerrit:settings/#HTTPCredentials|HTTP Credentials]], copy Username and click Generate new password to generate new password.
*In the old UI, go to [[gerrit:#/settings/http-password|HTTP Password]], copy Username and click Generate Password to generate new password.


*In the new UI go to [https://gerrit.wikimedia.org/r/settings/#SSHKeys SSH keys], copy the contents of your <code>~/.ssh/id_ed25519.pub</code> file into the ui.
{{note|type=error|Generated password in both cases is different from your Gerrit password.}}
*In the old UI, go to [https://gerrit.wikimedia.org/r/settings/ssh-keys#SSHKeys SSH keys], copy the contents of your <code>~/.ssh/id_ed25519.pub</code> file into the ui.


Create a new ssh-agent so that your key remains unlocked during branch cutting:
Make sure .netrc file is only readable by you.


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:~$ eval $(ssh-agent)
you@yourlaptop:~$ chmod go-rwx .netrc
USERNAME@deploy1001:~$ ssh-add ~/.ssh/id_ed25519
Enter passphrase for /home/USERNAME/.ssh/id_ed25519: <passphrase>
Identity added: /home/USERNAME/.ssh/id_ed25519 (USERNAME@deploy1001)
</syntaxhighlight>
</syntaxhighlight>


You should be able to list the keys in your agent and see your key there:
'''Clone or update <code>mediawiki/tools/release</code>.'''
 
<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:~$ ssh-add -l
USERNAME@yourlaptop:~$ git clone https://gerrit.wikimedia.org/r/mediawiki/tools/release
256 SHA256:WSjdx+WFnlo9Dd+FN63c35+Q3pArOD/TQpFEjNh7ODc USERNAME@deploy1001 (ED25519)
</syntaxhighlight>
</syntaxhighlight>


Add Gerrit to <code>~/.ssh/known_hosts</code>. Fingerprint is available only in the old UI at [https://gerrit.wikimedia.org/r/#/settings/ssh-keys settings/ssh-keys].
To run branch.py you need to have the pygerrit2 library installed for Python3. In Debian 10 (buster), the python3-pygerrit2 package works.
 
=====Create the new branch in Gerrit=====


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:~$ ssh -oFingerPrintHash=md5 -p 29418 GERRIT-USERNAME@gerrit.wikimedia.org
you@yourlaptop:~/release/make-release/ $ ./branch.py --core --core-bundle wmf_core --bundle wmf_branch --branchpoint HEAD --core-version [VERSION] [WMF BRANCH]
The authenticity of host '[gerrit.wikimedia.org]:29418 ([2620:0:861:3:208:80:154:85]:29418)' can't be established.
</syntaxhighlight>
RSA key fingerprint is MD5:dc:e9:68:7b:99:1b:27:d0:f9:fd:ce:6a:2e:bf:92:e1.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[gerrit.wikimedia.org]:29418,[2620:0:861:3:208:80:154:85]:29418' (RSA) to the list of known hosts.


  ****    Welcome to Gerrit Code Review    ****
<syntaxhighlight lang="shell-session">
 
you@yourlaptop:~/release/make-release/ $ ./branch.py --core --core-bundle wmf_core --bundle wmf_branch --branchpoint HEAD --core-version 1.34.0-wmf.0 wmf/1.34.0-wmf.0
  Hi You, you have successfully connected over SSH.
</syntaxhighlight>


  Unfortunately, interactive shells are disabled.
In {{irc|wikimedia-operations}}, drop a quick log note that you've kicked off the branch process so that others know it's underway, ''e.g.'':
  To clone a hosted Git repository, use:


  git clone ssh://GERRIT-USERNAME@gerrit.wikimedia.org:29418/REPOSITORY_NAME.git
<syntaxhighlight lang="irc">
 
!log 1.35.0-wmf.14 was branched at fb16374c5bdb9d14729f358fb81638fc91640b4f for T233862
Connection to gerrit.wikimedia.org closed.
</syntaxhighlight>
</syntaxhighlight>


Clone or update <code>mediawiki/tools/release</code>.
The script will create a release patch, [[gerrit:c/mediawiki/core/+/564687|like this one]], under your gerrit account. You must C+2 this, and wait for it to merge, to proceed.


<syntaxhighlight lang="shell-session">
=====tmux or screen =====
USERNAME@deploy1001:~$ git clone https://gerrit.wikimedia.org/r/mediawiki/tools/release
</syntaxhighlight>


====tmux or screen====
Now that the branch has been cut on your local machine, the remainder of the work will be done on the deployment host: '''deploy1002.eqiad.wmnet'''


Some scripts run for 10-60 minutes so consider using tmux or screen.
Some scripts run for 10-60 minutes so consider using tmux or screen.
Line 190: Line 236:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:~$ tmux new -s train
USERNAME@deploy1002:~$ tmux new -s train
...
...
USERNAME@deploy1001:~$ exit
USERNAME@deploy1002:~$ exit
</syntaxhighlight>
</syntaxhighlight>


Line 200: Line 246:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:~$ screen -D -RR train
USERNAME@deploy1002:~$ screen -D -RR train
...
...
USERNAME@deploy1001:~$ exit
USERNAME@deploy1002:~$ exit
</syntaxhighlight>
</syntaxhighlight>


If you need to leave in the middle you can do <code>ctrl-a d</code> to detach and <code>screen -r train</code> to attach.
If you need to leave in the middle you can do <code>ctrl-a d</code> to detach and <code>screen -r train</code> to attach.


====Create the new branch in Gerrit====
In either the tmux or the screen session, you'll want to start an ssh-agent and load your local key there:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:~/release/make-wmf-branch$ ./make-wmf-branch -n [VERSION] -o master
USERNAME@deploy1002:~$ eval $(ssh-agent)
USERNAME@deploy1002:~$ ssh-add .ssh/id_ed25519
</syntaxhighlight>
</syntaxhighlight>


<div style="margin:2em; padding: 1em; border:1px dashed silver;">
=====Clone new branch=====
'''DO NOT USE YET''': <code>branch.py</code> version:
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:~/release/make-release/ $ ./branch.py --core --core-bundle wmf_core --bundle wmf_branch --branchpoint HEAD --core-version [VERSION] [WMF BRANCH]
</syntaxhighlight>
</div>
 
Example:
 
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:~/release/make-wmf-branch$ ./make-wmf-branch -n 1.34.0-wmf.0 -o master
</syntaxhighlight>
 
<div style="margin:2em; padding: 1em; border:1px dashed silver;">
'''DO NOT USE YET''': <code>branch.py</code> version:
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:~/release/make-release/ $ ./branch.py --core --core-bundle wmf_core --bundle wmf_branch --branchpoint HEAD --core-version 1.34.0-wmf.0 wmf/1.34.0-wmf.0
</syntaxhighlight>
</div>
 
🐌 Note: the script will run for about 30 minutes.
 
If the process is interrupted (by rebooting Gerrit, for example), continue from where it stopped:
 
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:~/release/make-wmf-branch$ ./make-wmf-branch -n [VERSION] -o master -c extensions/[EXTENSION]
</syntaxhighlight>
 
====Clone new branch====
This command will create a new <code>/srv/mediawiki-staging/php-[VERSION]</code> directory:
This command will create a new <code>/srv/mediawiki-staging/php-[VERSION]</code> directory:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging$ scap prep [VERSION]
USERNAME@deploy1002:/srv/mediawiki-staging$ scap prep [VERSION]
</syntaxhighlight>
</syntaxhighlight>


Line 251: Line 270:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging$ scap prep 1.34.0-wmf.0
USERNAME@deploy1002:/srv/mediawiki-staging$ scap prep 1.34.0-wmf.0
</syntaxhighlight>
</syntaxhighlight>


This should only take a couple of minutes.
This should only take a couple of minutes.


====Apply security patches====
=====Apply security patches=====
 
; Short version : <code>scap patch [VERSION]</code>


*Patches should be named sequentially in the order that they will cleanly apply (e.g. <code>01-T[NUMBER].patch</code>, <code>02-T[NUMBER].patch</code>)
*Patches should be named sequentially in the order that they will cleanly apply (e.g. <code>01-T[NUMBER].patch</code>, <code>02-T[NUMBER].patch</code>)
*Check and apply each patch in both <code>/srv/patches/[VERSION]/core</code> and <code>/srv/patches/[VERSION]/extensions/[NAME]</code> to the new core checkout and extensions, respectively.
*Checks and applies each patch in both <code>/srv/patches/[VERSION]/core</code> and <code>/srv/patches/[VERSION]/extensions/[NAME]</code> to the new core checkout and extensions, respectively.


Check existing patches:
Check existing patches:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:~$ tree /srv/patches/[VERSION]
USERNAME@deploy1002:~$ tree /srv/patches/[VERSION]
/srv/patches/[VERSION]
/srv/patches/[VERSION]
├── core
├── core
Line 275: Line 296:
</syntaxhighlight>
</syntaxhighlight>


=====Core=====
======Core======


*You can check a core patch to see if it will apply cleanly with
*You can check a core patch to see if it will apply cleanly with


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git apply --check --3way /srv/patches/[VERSION]/core/[NUMBER]-T[NUMBER].patch
USERNAME@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git apply --check --3way /srv/patches/[VERSION]/core/[NUMBER]-T[NUMBER].patch
</syntaxhighlight>
</syntaxhighlight>


Line 286: Line 307:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/php-[VERSION]$ git am --3way /srv/patches/[VERSION]/core/[NUMBER]-T[NUMBER].patch
USERNAME@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git am --3way /srv/patches/[VERSION]/core/[NUMBER]-T[NUMBER].patch
</syntaxhighlight>
</syntaxhighlight>


=====Extension=====
======Extension======


* For an extension:
*For an extension:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/php-[VERSION]/extensions/[EXTENSION]$ git apply --check --3way /srv/patches/[VERSION]/extensions/[EXTENSION]/[NUMBER]-T[NUMBER].patch
USERNAME@deploy1002:/srv/mediawiki-staging/php-[VERSION]/extensions/[EXTENSION]$ git apply --check --3way /srv/patches/[VERSION]/extensions/[EXTENSION]/[NUMBER]-T[NUMBER].patch


USERNAME@deploy1001:/srv/mediawiki-staging/php-[VERSION]/extensions/[EXTENSION]$ git am --3way /srv/patches/[VERSION]/extensions/[EXTENSION]/[NUMBER]-T[NUMBER].patch
USERNAME@deploy1002:/srv/mediawiki-staging/php-[VERSION]/extensions/[EXTENSION]$ git am --3way /srv/patches/[VERSION]/extensions/[EXTENSION]/[NUMBER]-T[NUMBER].patch
</syntaxhighlight>
</syntaxhighlight>


*If the patch fails to apply, investigate whether it's due to a conflict (<code>git status</code>) or the patch having been merged since the new branch cut (search <code>git log</code> for the commit, etc.). If it turns out to be the latter, remove the patch file from the <code>/srv/patches/[VERSION]</code> directory.
*If the patch fails to apply, investigate whether it's due to a conflict (<code>git status</code>) or the patch having been merged since the new branch cut (search <code>git log</code> for the commit, etc.). If it turns out to be the latter, remove the patch file from the <code>/srv/patches/[VERSION]</code> directory.
*If you need extra help, contact Security Team ([https://wikimediafoundation.org/role/staff-contractors/ Wikimedia Foundation], [https://www.mediawiki.org/wiki/Wikimedia_Security_Team MediaWiki], [https://office.wikimedia.org/wiki/Contact_list#Security Office Wiki]), currently {{ircnick|bawolff|Brian}} and {{ircnick|Reedy|Sam}} in IRC.
*If you need extra help, contact Security Team ([[foundationsite:role/staff-contractors/|Wikimedia Foundation]], [[mw:Wikimedia_Security_Team|MediaWiki]], [https://office.wikimedia.org/wiki/Contact_list#Security Office Wiki]).


====Create patches to update wikiversions.json====
=====Create patches to update wikiversions.json=====


Create group0 to [VERSION] patch:
Create group0 to [VERSION] patch:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ scap update-wikiversions group0 [VERSION]
USERNAME@deploy1002:/srv/mediawiki-staging/$ scap update-wikiversions group0 [VERSION]
USERNAME@deploy1001:/srv/mediawiki-staging/$ git add wikiversions.json
USERNAME@deploy1002:/srv/mediawiki-staging/$ git add wikiversions.json
USERNAME@deploy1001:/srv/mediawiki-staging/$ git commit -m "Group0 to [VERSION]"
USERNAME@deploy1002:/srv/mediawiki-staging/$ git commit -m "Group0 to [VERSION]"
</syntaxhighlight>
</syntaxhighlight>


Line 315: Line 336:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ scap update-wikiversions group0 1.34.0-wmf.0
USERNAME@deploy1002:/srv/mediawiki-staging/$ scap update-wikiversions group0 1.34.0-wmf.0
USERNAME@deploy1001:/srv/mediawiki-staging/$ git add wikiversions.json
USERNAME@deploy1002:/srv/mediawiki-staging/$ git add wikiversions.json
USERNAME@deploy1001:/srv/mediawiki-staging/$ git commit -m "Group0 to 1.34.0-wmf.0"
USERNAME@deploy1002:/srv/mediawiki-staging/$ git commit -m "Group0 to 1.34.0-wmf.0"
</syntaxhighlight>
</syntaxhighlight>


====Send staged patches to Gerrit for review====
=====Send staged patches to Gerrit for review=====


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ git push origin HEAD:refs/for/master/[VERSION]
USERNAME@deploy1002:/srv/mediawiki-staging/$ git push origin HEAD:refs/for/master%topic=[VERSION]
</syntaxhighlight>
</syntaxhighlight>


Line 329: Line 350:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ git push origin HEAD:refs/for/master/1.34.0-wmf.0
USERNAME@deploy1002:/srv/mediawiki-staging/$ git push origin HEAD:refs/for/master%topic=1.34.0-wmf.0
</syntaxhighlight>
</syntaxhighlight>


====Discard changes to working directory and index====
=====Discard changes to working directory and index=====


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ git reset --hard origin/master
USERNAME@deploy1002:/srv/mediawiki-staging/$ git reset --hard origin/master
</syntaxhighlight>
</syntaxhighlight>


====Clean up old stuff====
=====Clean up old stuff=====


[[:mw:MediaWiki 1.34/Roadmap]] is a good place to find when a branch was created.
[[mw:MediaWiki 1.34/Roadmap]] is a good place to find when a branch was created.


List all branches:
List all branches:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ find . -maxdepth 1 -type d -name 'php-*' -print
USERNAME@deploy1002:/srv/mediawiki-staging/$ find . -maxdepth 1 -type d -name 'php-*' -print
</syntaxhighlight>
</syntaxhighlight>


Line 351: Line 372:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ find . -mindepth 2 -maxdepth 2 -type f -path './php-*/README' -ctime +7 -exec dirname {} \;
USERNAME@deploy1002:/srv/mediawiki-staging/$ find . -mindepth 2 -maxdepth 2 -type f -path './php-*/README.md' -ctime +7 -exec dirname {} \;
</syntaxhighlight>
</syntaxhighlight>


Line 357: Line 378:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ scap clean --delete [some old version from find -ctime +7 output above]
USERNAME@deploy1002:/srv/mediawiki-staging/$ scap clean --delete [some old version from find -ctime +7 output above]
</syntaxhighlight>
</syntaxhighlight>


Line 363: Line 384:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ scap clean --delete 1.34.0-wmf.0
USERNAME@deploy1002:/srv/mediawiki-staging/$ scap clean --delete 1.34.0-wmf.0
</syntaxhighlight>
</syntaxhighlight>


Active branches are visible at [https://tools.wmflabs.org/versions/ Wikimedia MediaWiki versions] page.
Active branches are visible at [https://versions.toolforge.org/ Wikimedia MediaWiki versions] page.


'''Deleting a branch is a full sync of that directory, and can take 10-15 minutes each.'''
'''Deleting a branch is a full sync of that directory, and can take 10-15 minutes each.'''


====Sync to cluster and verify on testwiki====
=====Sync to cluster and verify on testwiki =====


* Edit <code>/srv/mediawiki-staging/wikiversions.json</code> and set <code>testwiki</code> to <code>php-[VERSION]</code>
*Edit <code>/srv/mediawiki-staging/wikiversions.json</code> and set <code>testwiki</code> to <code>php-[VERSION]</code>
*Do not commit and push to Gerrit, only make this change locally on the deployment server
* Do not commit and push to Gerrit, only make this change locally on the deployment server


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ vim wikiversions.json
USERNAME@deploy1002:/srv/mediawiki-staging/$ vim wikiversions.json
</syntaxhighlight>
</syntaxhighlight>


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging$ git diff
USERNAME@deploy1002:/srv/mediawiki-staging$ git diff
...
...
-    "testwiki": "php-[VERSION-1]",
-    "testwiki": "php-[VERSION-1]",
Line 387: Line 408:
</syntaxhighlight>
</syntaxhighlight>


*Run [[scap]] to (re)build localization caches and sync changes across the cluster.
* Run [[scap]] to (re)build localization caches and sync changes across the cluster.
*🐌 Note: this step will take about 20 minutes.
*🐌 Note: this step may take on the order of 70-80 minutes.
* 🐌 Note: If scap seems to hang during the scap-cdb-rebuild phase of this process, you may need to try [[phab:T223287|pressing enter to speed it up]].


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ scap sync "testwiki to php-[VERSION] and rebuild l10n cache"
USERNAME@deploy1002:/srv/mediawiki-staging/$ scap sync "testwiki to php-[VERSION] and rebuild l10n cache"
</syntaxhighlight>
</syntaxhighlight>


Line 397: Line 419:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ scap sync "testwiki to php-1.34.0-wmf.0 and rebuild l10n cache"
USERNAME@deploy1002:/srv/mediawiki-staging/$ scap sync "testwiki to php-1.34.0-wmf.0 and rebuild l10n cache"
</syntaxhighlight>
</syntaxhighlight>


*Verify version change on [https://test.wikipedia.org/wiki/Special:Version testwiki] (Installed software, Product: MediaWiki, Version: [VERSION]) and l10n cache ([https://test.wikipedia.org/wiki/Special:Version Special:Version] should not look like [https://test.wikipedia.org/wiki/Special:Version?uselang=qqx Special:Version?uselang=qqx])
*Verify version change on [[testwiki:Special:Version|testwiki]] (Installed software, Product: MediaWiki, Version: [VERSION]) and l10n cache ([[testwiki:Special:Version|Special:Version]] should not look like [https://test.wikipedia.org/wiki/Special:Version?uselang=qqx Special:Version?uselang=qqx])


This can take half an hour. Opening or reloading the version page on testwiki after the scap sync command can take a minute or two.
This can take well over an hour. Opening or reloading the version page on testwiki after the scap sync command can take a minute or two.


*Revert local changes
*Revert local changes


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ git checkout -- wikiversions.json
USERNAME@deploy1002:/srv/mediawiki-staging/$ git checkout -- wikiversions.json
</syntaxhighlight>
</syntaxhighlight>


====Update deploy notes====
=====Update deploy notes=====
 
There is a script in the  [[gerrit:mediawiki/tools/release|<code>mediawiki/tools/release</code>]] repo called <code>makedeploynotes.py</code>. It queries gerrit's gitiles to make release notes. Usage: <code>makedeploynotes.py <old-version> <new-version></code>
 
<pre>
$ python3 make-deploy-notes/makedeploynotes.py 1.35.0-wmf.24 1.35.0-wmf.25 | tee deploy-notes-1.35.0-wmf.25
</pre>


*Deploy notes are automatically generated by the [https://integration.wikimedia.org/ci/job/train-deploy-notes Train Deploy Notes] Jenkins job after you cut the branch
From there you can copy-and-paste to the Changelog page on mediawiki.org; e.g., <code><nowiki>https://www.mediawiki.org/wiki/MediaWiki_[VERSION]/Changelog</nowiki></code>. Example: [[mw:MediaWiki_1.35/wmf.24/Changelog|MediaWiki 1.35/wmf.24/Changelog]]
*Be sure to check that the appropriate Changelog was created at <code><nowiki>https://www.mediawiki.org/wiki/MediaWiki_[VERSION]/Changelog</nowiki></code>. Example: [https://www.mediawiki.org/wiki/MediaWiki_1.34/wmf.4/Changelog MediaWiki 1.34/wmf.4/Changelog]


====Wait for deploy window====
=====Wait for deploy window=====
All of the changes above can be done at any time prior to the actual deployment window.
All of the changes above can be done at any time prior to the actual deployment window.
</div>


===During the deploy window===
====During the deploy window====


====Switch group0 wikis to [VERSION]====
;Short-form instructions
{| class="wikitable"
! colspan="2" |Step
!host
!command
!example
|-
|0-0
|'''Create and auto-merge/deploy the group0 patch'''
|deploy1002
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:/srv/mediawiki-staging/$ scap deploy-promote group0
Promote group0 from [PREVIOUS-VERSION] to [VERSION] [y/N]
Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions
</syntaxhighlight>
|-
|0-1
|'''Verify production has indeed switched'''
|[[mw:Special:Version|MediaWiki.org]]
| colspan="2" |Verify that [[mw:Special:Version|mediawikiwiki]] has switched to the new version (Installed software, Product: MediaWiki, Version: VERSION)
|-
| 0-2
|'''Monitor production logs'''
|logstash ''etc.''
| colspan="2" |Monitor irc and [[logstash]] and/or [[Wikimedia binaries#logspam-watch|logspam-watch]] for problems, see [[#Places to Watch for Breakage]]
|-
|0-3
|'''Update roadmap page'''
|[[mw:MediaWiki 1.39/Roadmap]]
|Change the <code>Deployed to group</code> (if you're using VisualEditor) or the 3rd parameter of the <code>WMFReleaseTableRow</code> template (if you're using the wikitext editor) to <code>0</code> (deployed to group0)
|<syntaxhighlight lang="text">
{{WMFReleaseTableHead}}
{{WMFReleaseTableRow|12|2018-07-10|0}}
</syntaxhighlight>
|-
|0-4
|'''Kill ssh-agent'''
|deployment server
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:~$ pgrep -u "$USER" -laf ssh-agent # list all of your ssh-agent processes
USERNAME@deploy1002:~$ pkill -u "$USER" -f ssh-agent  # kill all your ssh-agent processes
USERNAME@deploy1002:~$ pgrep -u "$USER" -laf ssh-agent # did they all die?</syntaxhighlight>
|}
 
;Full instructions
<div class="mw-collapsible mw-collapsed" style="margin:1em; border: 1px dashed grey; padding: 2em 1em;">
=====Switch group0 wikis to [VERSION] =====


*CR+2 <code>group0 to [VERSION]</code> patch in Gerrit that you submitted earlier
*CR+2 <code>group0 to [VERSION]</code> patch in Gerrit that you submitted earlier
*Wait for Gerrit/Zuul/Jenkins to merge the patch(es)
* Wait for Gerrit/Zuul/Jenkins to merge the patch(es)
* Pull patch(es) to deployment server
*Pull patch(es) to deployment server


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging$ git fetch
USERNAME@deploy1002:/srv/mediawiki-staging$ git fetch
</syntaxhighlight>
</syntaxhighlight>


Line 433: Line 507:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging$ git diff HEAD..origin/master
USERNAME@deploy1002:/srv/mediawiki-staging$ git diff HEAD..origin/master
</syntaxhighlight>
</syntaxhighlight>


Line 439: Line 513:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging$ git rebase origin/master
USERNAME@deploy1002:/srv/mediawiki-staging$ git rebase origin/master
</syntaxhighlight>
</syntaxhighlight>


Line 445: Line 519:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging$ scap sync-wikiversions "group0 to [VERSION]"
USERNAME@deploy1002:/srv/mediawiki-staging$ scap sync-wikiversions "group0 to [VERSION]"
</syntaxhighlight>
</syntaxhighlight>


Line 451: Line 525:


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging$ scap sync-wikiversions "group0 to 1.34.0-wmf.0"
USERNAME@deploy1002:/srv/mediawiki-staging$ scap sync-wikiversions "group0 to 1.34.0-wmf.0"
</syntaxhighlight>
</syntaxhighlight>


*Verify that [[:mw:Special:Version|mediawikiwiki]] switched to the new version (Installed software, Product: MediaWiki, Version: VERSION)
*Verify that [[:mw:Special:Version|mediawikiwiki]] switched to the new version (Installed software, Product: MediaWiki, Version: VERSION)
*Monitor irc and [[Logstash|logstash]] and/or [[Wikimedia_binaries#logspam-watch|logspam-watch]] for problems, see [[#Places to Watch for Breakage]]
*Monitor irc and [[logstash]] and/or [[Wikimedia_binaries#logspam-watch|logspam-watch]] for problems, see [[#Places to Watch for Breakage]]


====Update roadmap====
=====Update roadmap=====


*Change the <code>Deployed to group</code> (if you're using VisualEditor) or the 3rd parameter of the <code>WMFReleaseTableRow</code> template (if you're using the wikitext editor) to <code>0</code> (deployed to group0) at [[:mw:MediaWiki 1.35/Roadmap]].
*Change the <code>Deployed to group</code> (if you're using VisualEditor) or the 3rd parameter of the <code>WMFReleaseTableRow</code> template (if you're using the wikitext editor) to <code>0</code> (deployed to group0) at [[mw:MediaWiki 1.35/Roadmap]].


For wikitext editor, change
For wikitext editor, change
Line 488: Line 562:
</syntaxhighlight>
</syntaxhighlight>


==Wednesday: group0 to group1 deploy==
=====Terminate ssh-agents =====


==== Switch group1 wikis to [VERSION]====
Terminate the ssh-agent you started earlier so it doesn't linger on after you log out:


Use the <code>release/bin/deploy-promote</code> script to update <code>wikiversions.json</code>
<syntaxhighlight lang="shell-session">
pgrep -u "$USER" -laf ssh-agent # list all of your ssh-agent processes
pkill -u "$USER" -f ssh-agent  # kill all your ssh-agent processes
pgrep -u "$USER" -laf ssh-agent # did they all die?
</syntaxhighlight>
 
Every other day of the train you need to start a new ssh-agent and kill it later.
</div>
 
===Wednesday: group0 to group1 deploy===
 
;Meta / coordination
Attend the Train Log Triage meeting with members of the Core Platform Team and others.
 
;Short-form instructions
{| class="wikitable"
! colspan="2" |Step
!host
!command
!example
|-
|1-0
|'''Create and auto-merge/deploy the group1 patch'''
|deploy1001
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ scap deploy-promote group1
Promote group1 from [PREVIOUS-VERSION] to [VERSION] [y/N]
Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions
</syntaxhighlight>
|-
| 1-1
|'''Verify production has indeed switched'''
|[[wikt:Special:Version|English Wiktionary]]
| colspan="2" |Verify that [[wikt:Special:Version|the English Wiktionary]] (and other group1 wikis) have switched to the new version (Installed software, Product: MediaWiki, Version: VERSION)
|-
| 1-2
|'''Monitor production logs'''
|logstash ''etc.''
| colspan="2" |Monitor irc and [[logstash]] and/or [[Wikimedia binaries#logspam-watch|logspam-watch]] for problems, see [[#Places to Watch for Breakage]]
|-
|1-3
|'''Update roadmap page'''
|[[mw:MediaWiki 1.39/Roadmap]]
|Change the <code>Deployed to group</code> (if you're using VisualEditor) or the 3rd parameter of the <code>WMFReleaseTableRow</code> template (if you're using the wikitext editor) to <code>1</code> (deployed to group1)
|<syntaxhighlight lang="text">
{{WMFReleaseTableHead}}
{{WMFReleaseTableRow|12|2018-07-10|1}}
...
{{WMFReleaseTableFooter}}
</syntaxhighlight>
|}
 
;Full instructions
<div class="mw-collapsible mw-collapsed" style="margin:1em; border: 1px dashed grey; padding: 2em 1em;">
=====Switch group1 wikis to [VERSION]=====
 
Use <code>scap deploy-promote group1</code> to update <code>wikiversions.json</code>


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:~$ ./release/bin/deploy-promote
USERNAME@deploy1001:~$ scap deploy-promote group1
Promote group1 from [PREVIOUS-VERSION] to [VERSION] [y/N]
Promote group1 from [PREVIOUS-VERSION] to [VERSION] [y/N]
</syntaxhighlight>
</syntaxhighlight>
Line 509: Line 639:
The above should take about five minutes, including the waiting time for Gerrit/CI.
The above should take about five minutes, including the waiting time for Gerrit/CI.


====Update roadmap====
=====Update roadmap=====


*Change the <code>Deployed to group</code> (if you're using VisualEditor) or the 3rd parameter of the <code>WMFReleaseTableRow</code> template (if you're using the wikitext editor) to <code>1</code> (deployed to group1) at [[:mw:MediaWiki 1.35/Roadmap]].
*Change the <code>Deployed to group</code> (if you're using VisualEditor) or the 3rd parameter of the <code>WMFReleaseTableRow</code> template (if you're using the wikitext editor) to <code>1</code> (deployed to group1) at [[mw:MediaWiki 1.38/Roadmap]].


For wikitext editor, change
For wikitext editor, change
Line 530: Line 660:
{{WMFReleaseTableRow|12|2018-07-10|1}}
{{WMFReleaseTableRow|12|2018-07-10|1}}
</syntaxhighlight>
</syntaxhighlight>
</div>
===Thursday: group{0,1} to all deploy===
;Short-form instructions
{| class="wikitable"
!
!Step
!host
!command
!example
|-
|2-0
|'''Create and auto-merge/deploy the group2 patch'''
|deploy1001
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:/srv/mediawiki-staging/$ scap deploy-promote all
Promote all from [PREVIOUS-VERSION] to [VERSION] [y/N]
Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions
</syntaxhighlight>
|-
|2-1
|'''Verify production has indeed switched'''
|[[w:Special:Version|English Wikipedia]]
| colspan="2" |Verify that [[w:Special:Version|the English Wikipedia]] (and other group2 wikis) have switched to the new version (Installed software, Product: MediaWiki, Version: VERSION)
|-
| 2-2
|'''Monitor production logs'''
|logstash ''etc.''
| colspan="2" |Monitor irc and [[logstash]] and/or [[Wikimedia binaries#logspam-watch|logspam-watch]] for problems, see [[#Places to Watch for Breakage]]
|-
|2-3
|'''Update roadmap page'''
|[[mw:MediaWiki 1.39/Roadmap]]
|Change the <code>Deployed to group</code> (if you're using VisualEditor) or the 3rd parameter of the <code>WMFReleaseTableRow</code> template (if you're using the wikitext editor) to <code>2</code> (deployed to all)
|<syntaxhighlight lang="text">
{{WMFReleaseTableHead}}
{{WMFReleaseTableRow|12|2018-07-10|2}}
...
{{WMFReleaseTableFooter}}
</syntaxhighlight>
|}


==Thursday: group{0,1} to all deploy==
;Full instructions
<div class="mw-collapsible mw-collapsed" style="margin:1em; border: 1px dashed grey; padding: 2em 1em;">


==== Switch all wikis to [VERSION]====
=====Switch all wikis to [VERSION]=====


Thursday deploy is very similar to the Wednesday deploy, the only difference in terms of procedure is the target group
Thursday deploy is very similar to the Wednesday deploy, the only difference in terms of procedure is the target group


Use the <code>release/bin/deploy-promote all</code> script to update <code>wikiversions.json</code>
Use <code>scap deploy-promote all</code> to update <code>wikiversions.json</code>


<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
USERNAME@deploy1001:~$ ./release/bin/deploy-promote all
USERNAME@deploy1001:~$ scap deploy-promote all
Promote all from [PREVIOUS-VERSION] to [VERSION] [y/N]
Promote all from [PREVIOUS-VERSION] to [VERSION] [y/N]
</syntaxhighlight>
</syntaxhighlight>
Line 552: Line 724:
After the script run is complete, '''all wikis''' should be running [VERSION].
After the script run is complete, '''all wikis''' should be running [VERSION].


====Update roadmap====
=====Update roadmap=====


*Change the <code>Deployed to group</code> (if you're using VisualEditor) or the 3rd parameter of the <code>WMFReleaseTableRow</code> template (if you're using the wikitext editor) to <code>2</code> (deployed to all wikis) at [[:mw:MediaWiki 1.35/Roadmap]].
*Change the <code>Deployed to group</code> (if you're using VisualEditor) or the 3rd parameter of the <code>WMFReleaseTableRow</code> template (if you're using the wikitext editor) to <code>2</code> (deployed to all wikis) at [[mw:MediaWiki 1.38/Roadmap]].


For wikitext editor, change
For wikitext editor, change
Line 573: Line 745:
{{WMFReleaseTableRow|12|2018-07-10|2}}
{{WMFReleaseTableRow|12|2018-07-10|2}}
</syntaxhighlight>
</syntaxhighlight>
</div>


==Incident documentation==
==Incident documentation==
Line 578: Line 751:
*If there were problems during the train, follow instructions at [[Incident documentation]] on incident reports and post-mortem review.
*If there were problems during the train, follow instructions at [[Incident documentation]] on incident reports and post-mortem review.
*Use <code>Create report</code> form to create a new page, <code>train-[VERSION]</code>. Example: [[Incident documentation/20181212-Train-1.33.0-wmf.8]].
*Use <code>Create report</code> form to create a new page, <code>train-[VERSION]</code>. Example: [[Incident documentation/20181212-Train-1.33.0-wmf.8]].
*For Timeline section, events from [https://tools.wmflabs.org/sal/production SAL] and Phabricator task are a good start.
*For Timeline section, events from [https://sal.toolforge.org/production SAL] and Phabricator task are a good start.
 
== See also ==
*For information about the current status of the versions deployed to the various wikis, see https://versions.toolforge.org/
 
==Footnotes==
<references />


[[Category:How-To]]
[[Category:How-To]]
[[Category:Deployment]]
[[Category:Deployment]]

Latest revision as of 13:13, 10 June 2022

Deployments
Trainbows Not Painbows1.svg


Pairing on the Train

As of October 2019, there are two people assigned to each week's train: One as primary, and one as backup. These are rough guidelines for sharing the work, and should be improved as we learn more.

  • On Monday, communicate with your partner and establish how you'll collaborate over the course of the week.
    • Updates on IRC while your partner is working and updates on the train blocker ticket if they're offline seems to be a useful pattern.
    • Liberal use of video chat for pairing on hard problems is encouraged.
    • It seems to work well to have the primary do the work of cutting the branch, syncing wikis, etc., while the backup keeps an eye on logs, works on improvements to deploy tooling, and is generally an extra pair of eyes for the whole process.
    • If you are in doubt about any part of the process and it's during your partner's working hours, consult them first and get their help in resolving your questions.
  • If one member of the pair is in the European window and one is in the American window, both train deployment windows should be reserved on the Deployments calendar. This gives a backup deployer a defined window for moving the train forward outside the primary's working hours, if it becomes necessary.
  • If the train is blocked or there are any other issues, communicate the transfer of responsibility on the train blocker ticket by assigning it to the responsible party and leaving a note.

Initial setup

SSH into deploy1002.eqiad.wmnet and run

USERNAME@deploy1002:~$ git clone https://gerrit.wikimedia.org/r/mediawiki/tools/release

Breakage

There will be times when this process does not go smoothly. There are guidelines for what do to when that happens.

In general, if there is an unexplained error that occurs within 1 hour of a train deployment — always roll back the train. Rolling back the train to eliminate it as the cause of unexplained breakage can be especially important if there are many ongoing possible causes for issues as this helps to eliminate one of those causes as the source of problems.

Rollback

To rollback a wikiversion change, it should be pretty quick. Go ahead and rollback production before you send patches up to gerrit since waiting on Jenkins may take a while:

USERNAME@deploy1001:/srv/mediawiki-staging$ git revert $(git log -1 --format=%H -- wikiversions.json)
USERNAME@deploy1001:/srv/mediawiki-staging$ scap sync-wikiversions 'Revert "group[0|1] wikis to [VERSION]"'

# Now that you've synced the revert, push patches up to gerrit, you have to run git commit --amend to get the changeid
# Ideally, you should also add the train blocker task id to the Bug: field for this commit
USERNAME@deploy1001:/srv/mediawiki-staging$ git commit --amend
USERNAME@deploy1001:/srv/mediawiki-staging$ git push origin HEAD:refs/for/master%topic=[VERSION],l=Code-Review+2

Example:

USERNAME@deploy1001:/srv/mediawiki-staging$ git push origin HEAD:refs/for/master%topic=1.34.0-wmf.0,l=Code-Review+2
  • Wait for the patch to merge and the fetch back down to the deployment server

Places to Watch for Breakage

Train deployers should check for breakage as they are rolling out the train as they are effectively the first line of defense for train deploys. Some of the places to watch for breakage:

If the train is blocked

  • A task will be assigned to you, for example T191059 (1.32.0-wmf.13 deployment blockers) (you can see that week's task at https://train-blockers.toolforge.org)
  • Any open subtasks block the train from moving forward. This means no further deployments until the blockers are resolved.

Checklist

If there are blocking tasks, please do the following:

  • Make sure all tasks blocking train are set to UBN! priority in phabricator
  • Comment on the task asking for an ETA or if this can be solved by reverting a recent commit.
  • Send e-mail to:
    • ops@lists.wikimedia.org
    • wikitech-l@lists.wikimedia.org
    • Ping private #engineering-all Slack channel
    • Subject: [Train] {version} status update
    • Body
      The {version} version of MediaWiki is blocked[0].
      
      The new version is deployed to {group(s){0,1,2}}[1], but can proceed no
      further until these issues are resolved:
      
      * {Phab task name} - {phab task link}
      
      Once these issues are resolved train can resume. If these issues are
      resolved on a Friday the train will resume Monday.
      
      Thank you for your help resolving these issues!
      
      -- Your humble train toiler
      
      [0]. <{link to phab task for train}>
      [1]. <https://versions.toolforge.org/>
      
  • Add relevant people (see Developers/Maintainers) to the blocking task
  • Ping relevant people in IRC
  • Once train is unblocked be sure to thank the folks who helped unblock it

Weekly steps

Monday: Sync up with your deployment partner

See the train pairing section above.

Tuesday: New branch creation and deploy

Before the deploy window

Depending on how practiced you are and where you choose to run commands (full clones of mediawiki-core from outside the cluster can take a while), the steps will typically take 45 to 90 minutes.

Short-form instructions
Step host command example
P-0 Verify branch cut job worked Your laptop The branch cut is performed by a Jenkins job that runs on Tuesdays at 02:00 UTC on the releases-jenkins instance. The job creates and merges the wmf/* branch. It also builds and posts the changelog.

Navigate to Gerrit to find the branch commit that the job created. If there is no commit for the current train branch shown in Gerrit, you can troubleshoot via the releases-jenkins job.

P-3 Enter screen (or tmux if you prefer)


Note[1]

deploy1002.eqiad.wmnet
USERNAME@deploy1002:~$ screen -D -RR train
P-4 Set local ssh-agent in session deploy1002
USERNAME@deploy1002:~$ eval $(ssh-agent)
USERNAME@deploy1002:~$ ssh-add .ssh/id_ed25519
P-5 Clone new branch in production, apply patches, deploy to test-wikis, and cleanup old versions

🐌 Note: this step takes > 1 hour.

deploy1002
USERNAME@deploy1002:~$ scap stage-train [VERSION]
USERNAME@deploy1002:~$ scap stage-train 1.34.0-wmf.0

You can also run this command with the --dry-run option to see all the commands it will try to run:

USERNAME@deploy1002:~$ scap stage-train --dry-run 1.34.0-wmf.0
Wait for the deploy window
Full instructions
Setup

The script to cut a branch is run on your local machine (as of Jan 2020).

Local .netrc setup

Create a .netrc file in your home directory with the following content.

you@yourlaptop:~$ vim .netrc
machine gerrit.wikimedia.org login [USERNAME] password [PASSWORD]

Username and password can obtained from Gerrit:

  • In the new UI go to HTTP Credentials, copy Username and click Generate new password to generate new password.
  • In the old UI, go to HTTP Password, copy Username and click Generate Password to generate new password.

Make sure .netrc file is only readable by you.

you@yourlaptop:~$ chmod go-rwx .netrc

Clone or update mediawiki/tools/release.

USERNAME@yourlaptop:~$ git clone https://gerrit.wikimedia.org/r/mediawiki/tools/release

To run branch.py you need to have the pygerrit2 library installed for Python3. In Debian 10 (buster), the python3-pygerrit2 package works.

Create the new branch in Gerrit
you@yourlaptop:~/release/make-release/ $ ./branch.py --core --core-bundle wmf_core --bundle wmf_branch --branchpoint HEAD --core-version [VERSION] [WMF BRANCH]
you@yourlaptop:~/release/make-release/ $ ./branch.py --core --core-bundle wmf_core --bundle wmf_branch --branchpoint HEAD --core-version 1.34.0-wmf.0 wmf/1.34.0-wmf.0

In #wikimedia-operations connect, drop a quick log note that you've kicked off the branch process so that others know it's underway, e.g.:

!log 1.35.0-wmf.14 was branched at fb16374c5bdb9d14729f358fb81638fc91640b4f for T233862

The script will create a release patch, like this one, under your gerrit account. You must C+2 this, and wait for it to merge, to proceed.

tmux or screen

Now that the branch has been cut on your local machine, the remainder of the work will be done on the deployment host: deploy1002.eqiad.wmnet

Some scripts run for 10-60 minutes so consider using tmux or screen.

If you prefer tmux:

USERNAME@deploy1002:~$ tmux new -s train
...
USERNAME@deploy1002:~$ exit

If you need to leave in the middle you can do ctrl-b d to detach and tmux a -t train to attach.

If you prefer screen:

USERNAME@deploy1002:~$ screen -D -RR train
...
USERNAME@deploy1002:~$ exit

If you need to leave in the middle you can do ctrl-a d to detach and screen -r train to attach.

In either the tmux or the screen session, you'll want to start an ssh-agent and load your local key there:

USERNAME@deploy1002:~$ eval $(ssh-agent)
USERNAME@deploy1002:~$ ssh-add .ssh/id_ed25519
Clone new branch

This command will create a new /srv/mediawiki-staging/php-[VERSION] directory:

USERNAME@deploy1002:/srv/mediawiki-staging$ scap prep [VERSION]

Example:

USERNAME@deploy1002:/srv/mediawiki-staging$ scap prep 1.34.0-wmf.0

This should only take a couple of minutes.

Apply security patches
Short version
scap patch [VERSION]
  • Patches should be named sequentially in the order that they will cleanly apply (e.g. 01-T[NUMBER].patch, 02-T[NUMBER].patch)
  • Checks and applies each patch in both /srv/patches/[VERSION]/core and /srv/patches/[VERSION]/extensions/[NAME] to the new core checkout and extensions, respectively.

Check existing patches:

USERNAME@deploy1002:~$ tree /srv/patches/[VERSION]
/srv/patches/[VERSION]
├── core
│   ├── 01-T[NUMBER].patch
│   └── 02-T[NUMBER].patch
└── extensions
    └── [EXTENSION]
        ├── 01-T[NUMBER].patch
        └── 02-T[NUMBER].patch
Core
  • You can check a core patch to see if it will apply cleanly with
USERNAME@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git apply --check --3way /srv/patches/[VERSION]/core/[NUMBER]-T[NUMBER].patch
  • If the patch checks out, apply and commit it with
USERNAME@deploy1002:/srv/mediawiki-staging/php-[VERSION]$ git am --3way /srv/patches/[VERSION]/core/[NUMBER]-T[NUMBER].patch
Extension
  • For an extension:
USERNAME@deploy1002:/srv/mediawiki-staging/php-[VERSION]/extensions/[EXTENSION]$ git apply --check --3way /srv/patches/[VERSION]/extensions/[EXTENSION]/[NUMBER]-T[NUMBER].patch

USERNAME@deploy1002:/srv/mediawiki-staging/php-[VERSION]/extensions/[EXTENSION]$ git am --3way /srv/patches/[VERSION]/extensions/[EXTENSION]/[NUMBER]-T[NUMBER].patch
  • If the patch fails to apply, investigate whether it's due to a conflict (git status) or the patch having been merged since the new branch cut (search git log for the commit, etc.). If it turns out to be the latter, remove the patch file from the /srv/patches/[VERSION] directory.
  • If you need extra help, contact Security Team (Wikimedia Foundation, MediaWiki, Office Wiki).
Create patches to update wikiversions.json

Create group0 to [VERSION] patch:

USERNAME@deploy1002:/srv/mediawiki-staging/$ scap update-wikiversions group0 [VERSION]
USERNAME@deploy1002:/srv/mediawiki-staging/$ git add wikiversions.json
USERNAME@deploy1002:/srv/mediawiki-staging/$ git commit -m "Group0 to [VERSION]"

Example:

USERNAME@deploy1002:/srv/mediawiki-staging/$ scap update-wikiversions group0 1.34.0-wmf.0
USERNAME@deploy1002:/srv/mediawiki-staging/$ git add wikiversions.json
USERNAME@deploy1002:/srv/mediawiki-staging/$ git commit -m "Group0 to 1.34.0-wmf.0"
Send staged patches to Gerrit for review
USERNAME@deploy1002:/srv/mediawiki-staging/$ git push origin HEAD:refs/for/master%topic=[VERSION]

Example:

USERNAME@deploy1002:/srv/mediawiki-staging/$ git push origin HEAD:refs/for/master%topic=1.34.0-wmf.0
Discard changes to working directory and index
USERNAME@deploy1002:/srv/mediawiki-staging/$ git reset --hard origin/master
Clean up old stuff

mw:MediaWiki 1.34/Roadmap is a good place to find when a branch was created.

List all branches:

USERNAME@deploy1002:/srv/mediawiki-staging/$ find . -maxdepth 1 -type d -name 'php-*' -print

Find old branches, more than 7 days old:

USERNAME@deploy1002:/srv/mediawiki-staging/$ find . -mindepth 2 -maxdepth 2 -type f -path './php-*/README.md' -ctime +7 -exec dirname {} \;

For all branches more than 7 days old, drop everything with:

USERNAME@deploy1002:/srv/mediawiki-staging/$ scap clean --delete [some old version from find -ctime +7 output above]

Example:

USERNAME@deploy1002:/srv/mediawiki-staging/$ scap clean --delete 1.34.0-wmf.0

Active branches are visible at Wikimedia MediaWiki versions page.

Deleting a branch is a full sync of that directory, and can take 10-15 minutes each.

Sync to cluster and verify on testwiki
  • Edit /srv/mediawiki-staging/wikiversions.json and set testwiki to php-[VERSION]
  • Do not commit and push to Gerrit, only make this change locally on the deployment server
USERNAME@deploy1002:/srv/mediawiki-staging/$ vim wikiversions.json
USERNAME@deploy1002:/srv/mediawiki-staging$ git diff
...
-    "testwiki": "php-[VERSION-1]",
+    "testwiki": "php-[VERSION]",
...
  • Run scap to (re)build localization caches and sync changes across the cluster.
  • 🐌 Note: this step may take on the order of 70-80 minutes.
  • 🐌 Note: If scap seems to hang during the scap-cdb-rebuild phase of this process, you may need to try pressing enter to speed it up.
USERNAME@deploy1002:/srv/mediawiki-staging/$ scap sync "testwiki to php-[VERSION] and rebuild l10n cache"

Example:

USERNAME@deploy1002:/srv/mediawiki-staging/$ scap sync "testwiki to php-1.34.0-wmf.0 and rebuild l10n cache"

This can take well over an hour. Opening or reloading the version page on testwiki after the scap sync command can take a minute or two.

  • Revert local changes
USERNAME@deploy1002:/srv/mediawiki-staging/$ git checkout -- wikiversions.json
Update deploy notes

There is a script in the mediawiki/tools/release repo called makedeploynotes.py. It queries gerrit's gitiles to make release notes. Usage: makedeploynotes.py <old-version> <new-version>

$ python3 make-deploy-notes/makedeploynotes.py 1.35.0-wmf.24 1.35.0-wmf.25 | tee deploy-notes-1.35.0-wmf.25

From there you can copy-and-paste to the Changelog page on mediawiki.org; e.g., https://www.mediawiki.org/wiki/MediaWiki_[VERSION]/Changelog. Example: MediaWiki 1.35/wmf.24/Changelog

Wait for deploy window

All of the changes above can be done at any time prior to the actual deployment window.

During the deploy window

Short-form instructions
Step host command example
0-0 Create and auto-merge/deploy the group0 patch deploy1002
USERNAME@deploy1002:/srv/mediawiki-staging/$ scap deploy-promote group0
Promote group0 from [PREVIOUS-VERSION] to [VERSION] [y/N]
Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions
0-1 Verify production has indeed switched MediaWiki.org Verify that mediawikiwiki has switched to the new version (Installed software, Product: MediaWiki, Version: VERSION)
0-2 Monitor production logs logstash etc. Monitor irc and logstash and/or logspam-watch for problems, see #Places to Watch for Breakage
0-3 Update roadmap page mw:MediaWiki 1.39/Roadmap Change the Deployed to group (if you're using VisualEditor) or the 3rd parameter of the WMFReleaseTableRow template (if you're using the wikitext editor) to 0 (deployed to group0)
{{WMFReleaseTableHead}}
{{WMFReleaseTableRow|12|2018-07-10|0}}
0-4 Kill ssh-agent deployment server
USERNAME@deploy1002:~$ pgrep -u "$USER" -laf ssh-agent # list all of your ssh-agent processes
USERNAME@deploy1002:~$ pkill -u "$USER" -f ssh-agent   # kill all your ssh-agent processes
USERNAME@deploy1002:~$ pgrep -u "$USER" -laf ssh-agent # did they all die?
Full instructions
Switch group0 wikis to [VERSION]
  • CR+2 group0 to [VERSION] patch in Gerrit that you submitted earlier
  • Wait for Gerrit/Zuul/Jenkins to merge the patch(es)
  • Pull patch(es) to deployment server
USERNAME@deploy1002:/srv/mediawiki-staging$ git fetch
  • Check diff to ensure it is what you expect (this should show a bunch of version changes in wikiversions.json for group0 wikis)
USERNAME@deploy1002:/srv/mediawiki-staging$ git diff HEAD..origin/master
  • Apply changes
USERNAME@deploy1002:/srv/mediawiki-staging$ git rebase origin/master
  • Sync the change across the cluster
USERNAME@deploy1002:/srv/mediawiki-staging$ scap sync-wikiversions "group0 to [VERSION]"

Example:

USERNAME@deploy1002:/srv/mediawiki-staging$ scap sync-wikiversions "group0 to 1.34.0-wmf.0"
Update roadmap
  • Change the Deployed to group (if you're using VisualEditor) or the 3rd parameter of the WMFReleaseTableRow template (if you're using the wikitext editor) to 0 (deployed to group0) at mw:MediaWiki 1.35/Roadmap.

For wikitext editor, change

{{WMFReleaseTableHead}}
{{WMFReleaseTableRow|[VERSION]|[DATE]|}}
...
{{WMFReleaseTableFooter}}

to

{{WMFReleaseTableHead}}
{{WMFReleaseTableRow|[VERSION]|[DATE]|0}}
...
{{WMFReleaseTableFooter}}

Example:

{{WMFReleaseTableHead}}
{{WMFReleaseTableRow|12|2018-07-10|0}}
...
{{WMFReleaseTableFooter}}
Terminate ssh-agents

Terminate the ssh-agent you started earlier so it doesn't linger on after you log out:

pgrep -u "$USER" -laf ssh-agent # list all of your ssh-agent processes
pkill -u "$USER" -f ssh-agent   # kill all your ssh-agent processes
pgrep -u "$USER" -laf ssh-agent # did they all die?

Every other day of the train you need to start a new ssh-agent and kill it later.

Wednesday: group0 to group1 deploy

Meta / coordination

Attend the Train Log Triage meeting with members of the Core Platform Team and others.

Short-form instructions
Step host command example
1-0 Create and auto-merge/deploy the group1 patch deploy1001
USERNAME@deploy1001:/srv/mediawiki-staging/$ scap deploy-promote group1
Promote group1 from [PREVIOUS-VERSION] to [VERSION] [y/N]
Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions
1-1 Verify production has indeed switched English Wiktionary Verify that the English Wiktionary (and other group1 wikis) have switched to the new version (Installed software, Product: MediaWiki, Version: VERSION)
1-2 Monitor production logs logstash etc. Monitor irc and logstash and/or logspam-watch for problems, see #Places to Watch for Breakage
1-3 Update roadmap page mw:MediaWiki 1.39/Roadmap Change the Deployed to group (if you're using VisualEditor) or the 3rd parameter of the WMFReleaseTableRow template (if you're using the wikitext editor) to 1 (deployed to group1)
{{WMFReleaseTableHead}}
{{WMFReleaseTableRow|12|2018-07-10|1}}
...
{{WMFReleaseTableFooter}}
Full instructions
Switch group1 wikis to [VERSION]

Use scap deploy-promote group1 to update wikiversions.json

USERNAME@deploy1001:~$ scap deploy-promote group1
Promote group1 from [PREVIOUS-VERSION] to [VERSION] [y/N]

The script automatically Code-Review +2 the patch in Gerrit. Once CI has merged the patch, hit enter at the 2nd prompt.

Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions

After the script run is complete, group1 wikis should be running [VERSION].

The above should take about five minutes, including the waiting time for Gerrit/CI.

Update roadmap
  • Change the Deployed to group (if you're using VisualEditor) or the 3rd parameter of the WMFReleaseTableRow template (if you're using the wikitext editor) to 1 (deployed to group1) at mw:MediaWiki 1.38/Roadmap.

For wikitext editor, change

{{WMFReleaseTableRow|[VERSION]|[DATE]|0}}

to

{{WMFReleaseTableRow|[VERSION]|[DATE]|1}}

Example:

{{WMFReleaseTableRow|12|2018-07-10|1}}

Thursday: group{0,1} to all deploy

Short-form instructions
Step host command example
2-0 Create and auto-merge/deploy the group2 patch deploy1001
USERNAME@deploy1001:/srv/mediawiki-staging/$ scap deploy-promote all
Promote all from [PREVIOUS-VERSION] to [VERSION] [y/N]
Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions
2-1 Verify production has indeed switched English Wikipedia Verify that the English Wikipedia (and other group2 wikis) have switched to the new version (Installed software, Product: MediaWiki, Version: VERSION)
2-2 Monitor production logs logstash etc. Monitor irc and logstash and/or logspam-watch for problems, see #Places to Watch for Breakage
2-3 Update roadmap page mw:MediaWiki 1.39/Roadmap Change the Deployed to group (if you're using VisualEditor) or the 3rd parameter of the WMFReleaseTableRow template (if you're using the wikitext editor) to 2 (deployed to all)
{{WMFReleaseTableHead}}
{{WMFReleaseTableRow|12|2018-07-10|2}}
...
{{WMFReleaseTableFooter}}
Full instructions
Switch all wikis to [VERSION]

Thursday deploy is very similar to the Wednesday deploy, the only difference in terms of procedure is the target group

Use scap deploy-promote all to update wikiversions.json

USERNAME@deploy1001:~$ scap deploy-promote all
Promote all from [PREVIOUS-VERSION] to [VERSION] [y/N]

The script automatically Code-Review +2 the patch in Gerrit. Once CI has merged the patch, hit enter at the 2nd prompt.

Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions

After the script run is complete, all wikis should be running [VERSION].

Update roadmap
  • Change the Deployed to group (if you're using VisualEditor) or the 3rd parameter of the WMFReleaseTableRow template (if you're using the wikitext editor) to 2 (deployed to all wikis) at mw:MediaWiki 1.38/Roadmap.

For wikitext editor, change

{{WMFReleaseTableRow|[VERSION]|[DATE]|1}}

to

{{WMFReleaseTableRow|[VERSION]|[DATE]|2}}

Example:

{{WMFReleaseTableRow|12|2018-07-10|2}}

Incident documentation

See also

Footnotes

  1. If you need to leave in the middle you can do ctrl-a d to detach and screen -r train to attach.