View source for Release Engineering/Drafts/Deployments/How to deploy the train

{{draft}}
{{ptag|Train Deployments}}

[[File:MTR_CSR_Sifang_EMU_in_Shek_Kong_Stabling_Sidings_201710.jpg|thumb|500x500px|Bring new code in a fast, safe and efficient way!]]
{{Navigation MediaWiki deployment}}

== Initial setup ==

If this is your first time running the train, you need to do some initial configuration.  Start by SSHing into '''deploy1002.eqiad.wmnet'''.

First, check out the <code>mediawiki/tools/release</code> repo:

<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:~$ git clone https://gerrit.wikimedia.org/r/mediawiki/tools/release
</syntaxhighlight>

Next, make sure you're using a full-featured Git prompt for Bash, by adding the following to your <code>~/.profile</code>:

<syntaxhighlight lang="shell">
GIT_PS1_SHOWUNTRACKEDFILES=1
GIT_PS1_SHOWDIRTYSTATE=1
GIT_PS1_SHOWUPSTREAM="auto verbose"
. /etc/bash_completion.d/git-prompt
PS1='\u@\h \w$(__git_ps1 " (%s)") \$ '
</syntaxhighlight>

== Breakage ==

There will be times when this process does not go smoothly. There are [[Deployments/Holding_the_train|guidelines]] for what do to when that happens.

In general, '''if there is an unexplained error that occurs within 1 hour of a train deployment — always roll back the train'''. Rolling back the train to eliminate it as the cause of unexplained breakage can be especially important if there are many ongoing possible causes for issues as this helps to eliminate one of those causes as the source of problems.

=== Rollback ===

It should be quick to roll back wikiversion changes. Rollback production before you send patches to Gerrit since waiting on CI may take a while:

<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:/srv/mediawiki-staging$ git revert $(git log -1 --format=%H -- wikiversions.json)
USERNAME@deploy1002:/srv/mediawiki-staging$ scap sync-wikiversions 'Revert "group[0|1] wikis to [VERSION]"'
USERNAME@deploy1002:/srv/mediawiki-staging$ # Now that you've synced the revert, push patches to gerrit. You have to run git commit --amend to get the changeid:
USERNAME@deploy1002:/srv/mediawiki-staging$ git commit --amend
USERNAME@deploy1002:/srv/mediawiki-staging$ git push origin HEAD:refs/for/master%topic=[VERSION],l=Code-Review+2
</syntaxhighlight>

Example:

<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:/srv/mediawiki-staging$ git push origin HEAD:refs/for/master%topic=1.34.0-wmf.0,l=Code-Review+2
</syntaxhighlight>

* Wait for the patch to merge and then fetch back down to the deployment server

* [[#Update roadmap]].

=== Places to Watch for Breakage ===

Train deployers should check for breakage as they are rolling out the train, as they are effectively the first line of defense for train deploys. Some of the places to watch for breakage:

* IRC
** Primary channel is {{irc|wikimedia-operations}}
** Useful channels are {{irc|mediawiki-core}} {{irc|wikimedia-dev}}
** For more channels see [[mw:MediaWiki_on_IRC|MediaWiki on IRC]] and [[metawiki:IRC/Channels|IRC/Channels]]
* Logs
** [[mwlog1001]]:
*** [[Wikimedia_binaries#logspam-watch|logspam-watch]]
*** logfiles in <code>/srv/mw-log</code>
** [https://logstash.wikimedia.org/app/kibana#/dashboard/mediawiki-errors Logstash MediaWiki Errors]
** Logstash "mediawiki-new-errors" dashboard (linked from logstash front page)
*** [https://logstash.wikimedia.org/app/kibana#/dashboard/dfcf7b70-1aaa-11e9-b4bc-db12fe15ab31 Showing only timeout errors] (see T204871)
** Group-specific Logstash Dashboards:
*** [https://logstash.wikimedia.org/app/kibana#/dashboard/group0 group0]
*** [https://logstash.wikimedia.org/app/kibana#/dashboard/group1 group1]
** See the [[phab:tag/wikimedia-production-error/|Wikimedia-production-error workboard]] for known issues
** Logstash [https://logstash.wikimedia.org/app/dashboards#/view/AXDBY8Qhh3Uj6x1zCF56 mw-client-errors] dashboard
*** New errors appearing more than 1000 times in a 12 hour period should be considered blockers
*** See also [https://grafana.wikimedia.org/d/000000566/overview?viewPanel=16&orgId=1 Grafana dashboard] with summary of average error rate over time
* [https://grafana.wikimedia.org/ Grafana]
** [https://grafana.wikimedia.org/dashboard/db/varnish-http-errors?refresh=5m&orgId=1 Varnish error-rate dashboard] (HTTP 5XX % should have 3+ 0s after the decimal point, e.g. 0.0001%)
** [https://grafana.wikimedia.org/d/000000612/frontend-responses-nginx-vs-varnish?orgId=1&from=now-15m&to=now Frontend Responses NGINX vs Varnish]
** [https://grafana.wikimedia.org/d/000000102/production-logging Production Logging]
** [https://grafana.wikimedia.org/d/000000566/overview?panelId=15&fullscreen&orgId=1&from=now-7d&to=now Minerva Client Errors] - Browser JS errors count (only wikipedias on mobile)
** [https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?orgId=1&from=now-1h&to=now&refresh=30s Application Servers RED Dashboard]

=== If the train is blocked ===

* A task will be assigned to you, for example [[phab:T191059|T191059]] (1.32.0-wmf.13 deployment blockers) (you can see that week's task at https://train-blockers.toolforge.org)
* Any open subtasks block the train from moving forward. This means no further deployments until the blockers are resolved.

'''Checklist'''

If there are blocking tasks, please do the following:

* Make sure all tasks blocking train are set to <code>Unbreak Now!</code> priority in phabricator
* Comment on the task asking for an ETA or if this can be solved by reverting a recent commit.
* Send e-mail to:
** [[mail:ops|ops@lists.wikimedia.org]]
** [[mail:wikitech-l|wikitech-l@lists.wikimedia.org]]
** Ping private [https://app.slack.com/client/T024KLHS4/C01R06P8D1B/ #engineering-all Slack channel]
** Subject: <code>[Train] {version} status update: {brief summary}</code>
** Body<syntaxhighlight lang="text">The {version} version of MediaWiki is blocked[0].

The new version is deployed to {group(s){0,1,2}}[1], but can proceed no
further until these issues are resolved:

* {Phab task name} - {phab task link}

Once these issues are resolved train can resume. If these issues are
resolved on a Friday the train will resume Monday.

Thank you for your help resolving these issues!

-- Your humble train toiler

[0]. <{link to phab task for train}>
[1]. <https://versions.toolforge.org/></syntaxhighlight>
* Tag relevant teams and people (see [[mw:Developers/Maintainers|Developers/Maintainers]]) on the blocking task
* Ping relevant people in IRC
* Once train is unblocked be sure to thank the folks who helped unblock it

== Weekly steps ==

=== Monday: Sync up with your deployment partner===

There are two people assigned to each week's train:  One as primary, and one as backup.

The primary train conductor will be the assignee of the train blocker task in Phabricator.  Backup conductor will be listed as Backup Conductor at the top of the task.  On Monday, you should communicate briefly with your partner and establish how you'll collaborate over the course of the week.

See [[Release Engineering/Drafts/Deployments/How to pair on the train]] for an overview of helpful practices.

=== Tuesday: New branch creation and deploy ===

==== Before the deploy window ====

Depending on how practiced you are and where you choose to run commands (full clones of mediawiki-core from outside the cluster can take a while), the steps will typically take 45 to 90 minutes.

; Short-form instructions
{| class="wikitable"
! colspan="2" |Step
!host
!command
!example
|-
| P-0
|'''Verify branch cut job worked'''
|Your laptop
| colspan="2" |The branch cut runs in a periodic jenkins job that runs on Tuesdays at 02:00 UTC on the releases-jenkins instance. Navigate to [[gerrit:#/q/owner:%22TrainBranchBot+%253Cmmodell%252Btrainbranchbot%2540wikimedia.org%253E%22+status:open|gerrit]] to find the branch commit that the job created.

If there are no open commits shown in gerrit using the link above, you can troubleshoot via the [https://releases-jenkins.wikimedia.org/job/Automatic%20branch%20cut/ releases-jenkins job].
<br>
This will also build and post the changelog for you.
|-
|P-1
|'''Note the MW core commit from which you've just created the branch'''
|IRC
(<code>#wikimedia-operations)</code>
|<syntaxhighlight lang="irc">
!log [VERSION] was branched at [BRANCH POINT] for [TASK]
</syntaxhighlight>
|<syntaxhighlight lang="irc">
!log 1.35.0-wmf.14 was branched at fb16374c5bdb9d14729f358fb81638fc91640b4f for T233862
</syntaxhighlight>
|-
|P-2
|'''Merge the branch commit'''
|Gerrit ([[gerrit:c/mediawiki/core/+/595915|example]])
| colspan="2" |C+2 on the patch.  It takes about 25 minutes for the branch to be tested and merged.
|-
|P-3
|'''Enter screen (or tmux if you prefer)'''


Note<ref>If you need to leave in the middle you can do <code>ctrl-a d</code> to detach and <code>screen -r train</code> to attach.</ref>
|'''deploy1002.eqiad.wmnet'''
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:~$ screen -D -RR train
</syntaxhighlight>
|-
|P-4
|'''Set local ssh-agent in session'''
|deploy1002
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:~$ eval $(ssh-agent)
USERNAME@deploy1002:~$ ssh-add .ssh/id_ed25519
</syntaxhighlight>
|-
|P-5
|'''Clone new branch in production''' (once the branch commit from P-2 has landed)
|deploy1002
|<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:/srv/mediawiki-staging$ scap prep [VERSION]
</syntaxhighlight>
|<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:/srv/mediawiki-staging$ scap prep 1.34.0-wmf.0
</syntaxhighlight>
|-
|P-6
|'''Apply security patches'''
|deploy1002
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:/srv/mediawiki-staging$ scap apply-patches --train [VERSION]
</syntaxhighlight>

* See https://phabricator.wikimedia.org/T276237 for information about currently deploy security patches.
* If a patch fails to apply, investigate whether it's due to a conflict (<code>git status</code>) or the patch having been merged since the new branch cut (search <code>git log</code> for the commit, etc.). If it turns out to be the latter, remove the patch file from the <code>/srv/patches/[VERSION]</code> directory.
*If you need extra help, contact Security Team ([[foundationsite:role/staff-contractors/|Wikimedia Foundation]], [[Wikimedia Security Team|MediaWiki]], [https://office.wikimedia.org/wiki/Contact_list#Security Office Wiki]), currently {{ircnick|bawolff|Brian}} and {{ircnick|Reedy|Sam}} in IRC.
|-
|P-7
|'''Create and auto-merge/deploy the testwikis patch'''


🐌 Note: this step takes about 40 minutes.
|deploy1002
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:/srv/mediawiki-staging/$ ~/release/bin/deploy-promote testwikis [VERSION]
Promote testwikis from [PREVIOUS-VERSION] to [VERSION] [y/N]
Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions
</syntaxhighlight>
|-
|P-8
|'''Verify version change on [[testwiki:Special:Version|testwiki]]'''
|[[testwiki:Special:Version|testwiki]]
| colspan="2" |Verify version change on [[testwiki:Special:Version|testwiki]] (Installed software, Product: MediaWiki, Version: [VERSION]) and l10n cache ([[testwiki:Special:Version|Special:Version]] should not look like [https://test.wikipedia.org/wiki/Special:Version?uselang=qqx Special:Version?uselang=qqx]).<br>This is done automatically by the deploy script, but is worth verifying manually.
|-
|P-9
|'''Decide what old stuff to prune'''
|deploy1002
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:/srv/mediawiki-staging/$ find . -mindepth 2 -maxdepth 2 -type f -path './php-*/README.md' -ctime +7 -exec dirname {} \;
</syntaxhighlight>
|-
|P-10
|'''Clean up old stuff'''

🐌 Note: this step runs a scap sync of the directory, and can take 30 minutes.
|deploy1002
|<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:/srv/mediawiki-staging/$ scap clean --delete [some old version from find -ctime +7 output above]
</syntaxhighlight>
|<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:/srv/mediawiki-staging/$ scap clean --delete 1.34.0-wmf.0
</syntaxhighlight>
|-
! colspan="5" |'''Wait for the deploy window'''
|}

====During the deploy window====

;Short-form instructions
{| class="wikitable"
! colspan="2" |Step
!host
!command
!example
|-
|0-0
|'''Create and auto-merge/deploy the group0 patch'''
|deploy1002
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:/srv/mediawiki-staging/$ ~/release/bin/deploy-promote group0
Promote group0 from [PREVIOUS-VERSION] to [VERSION] [y/N]
Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions
</syntaxhighlight>
|-
|0-1
|'''Verify production has indeed switched'''
|[[mw:Special:Version|MediaWiki.org]]
| colspan="2" |Verify that [[mw:Special:Version|mediawikiwiki]] has switched to the new version (Installed software, Product: MediaWiki, Version: VERSION)
|-
| 0-2
|'''Monitor production logs'''
|logstash ''etc.''
| colspan="2" |Monitor irc and [[logstash]] and/or [[Wikimedia binaries#logspam-watch|logspam-watch]] for problems, see [[#Places to Watch for Breakage]]
|-
|0-3
|'''Update roadmap page'''
|[[mw:MediaWiki 1.36/Roadmap]]
|Change the <code>Deployed to group</code> (if you're using VisualEditor) or the 3rd parameter of the <code>WMFReleaseTableRow</code> template (if you're using the wikitext editor) to <code>0</code> (deployed to group0)
|<syntaxhighlight lang="text">
{{WMFReleaseTableHead}}
{{WMFReleaseTableRow|12|2018-07-10|0}}
</syntaxhighlight>
|-
|0-4
|'''Kill ssh-agent'''
|deployment server
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:~$ pgrep -u "$USER" -laf ssh-agent # list all of your ssh-agent processes
USERNAME@deploy1002:~$ pkill -u "$USER" -f ssh-agent   # kill all your ssh-agent processes
USERNAME@deploy1002:~$ pgrep -u "$USER" -laf ssh-agent # did they all die?</syntaxhighlight>
|}

===Wednesday: group0 to group1 deploy===

;Meta / coordination
Attend the Train Log Triage meeting with members of the Core Platform Team and others.

;Short-form instructions
{| class="wikitable"
! colspan="2" |Step
!host
!command
!example
|-
|1-0
|'''Create and auto-merge/deploy the group1 patch'''
|deploy1002
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:/srv/mediawiki-staging/$ ~/release/bin/deploy-promote
Promote group1 from [PREVIOUS-VERSION] to [VERSION] [y/N]
Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions
</syntaxhighlight>
|-
| 1-1
|'''Verify production has indeed switched'''
|[[wikt:Special:Version|English Wiktionary]]
| colspan="2" |Verify that [[wikt:Special:Version|the English Wiktionary]] (and other group1 wikis) have switched to the new version (Installed software, Product: MediaWiki, Version: VERSION)
|-
| 1-2
|'''Monitor production logs'''
|logstash ''etc.''
| colspan="2" |Monitor irc and [[logstash]] and/or [[Wikimedia binaries#logspam-watch|logspam-watch]] for problems, see [[#Places to Watch for Breakage]]
|-
|1-3
|'''Update roadmap page'''
|[[mw:MediaWiki 1.36/Roadmap]]
|Change the <code>Deployed to group</code> (if you're using VisualEditor) or the 3rd parameter of the <code>WMFReleaseTableRow</code> template (if you're using the wikitext editor) to <code>1</code> (deployed to group1)
|<syntaxhighlight lang="text">
{{WMFReleaseTableHead}}
{{WMFReleaseTableRow|12|2018-07-10|1}}
...
{{WMFReleaseTableFooter}}
</syntaxhighlight>
|}

===Thursday: group{0,1} to all deploy===
;Short-form instructions
{| class="wikitable"
!
!Step
!host
!command
!example
|-
|2-0
|'''Create and auto-merge/deploy the group2 patch'''
|deploy1002
| colspan="2" |<syntaxhighlight lang="shell-session">
USERNAME@deploy1002:/srv/mediawiki-staging/$ ~/release/bin/deploy-promote all
Promote all from [PREVIOUS-VERSION] to [VERSION] [y/N]
Now wait for jenkins to merge the patch, then press enter to continue with git pull && scap sync-wikiversions
</syntaxhighlight>
|-
|2-1
|'''Verify production has indeed switched'''
|[[w:Special:Version|English Wikipedia]]
| colspan="2" |Verify that [[w:Special:Version|the English Wikipedia]] (and other group2 wikis) have switched to the new version (Installed software, Product: MediaWiki, Version: VERSION)
|-
| 2-2
|'''Monitor production logs'''
|logstash ''etc.''
| colspan="2" |Monitor irc and [[logstash]] and/or [[Wikimedia binaries#logspam-watch|logspam-watch]] for problems, see [[#Places to Watch for Breakage]]
|-
|2-3
|'''Update roadmap page'''
|[[mw:MediaWiki 1.36/Roadmap]]
|Change the <code>Deployed to group</code> (if you're using VisualEditor) or the 3rd parameter of the <code>WMFReleaseTableRow</code> template (if you're using the wikitext editor) to <code>2</code> (deployed to all)
|<syntaxhighlight lang="text">
{{WMFReleaseTableHead}}
{{WMFReleaseTableRow|12|2018-07-10|2}}
...
{{WMFReleaseTableFooter}}
</syntaxhighlight>
|}

== Incident documentation ==

* If there were problems during the train, follow instructions at [[Incident documentation]] on incident reports and post-mortem review.
* Use <code>Create report</code> form to create a new page, <code>train-[VERSION]</code>. Example: [[Incident documentation/20181212-Train-1.33.0-wmf.8]].
* For the Timeline section, events from [https://sal.toolforge.org/production SAL] and Phabricator task are a good start.

== See also ==
* For the current versions deployed to the various wikis, see https://versions.toolforge.org/

== Footnotes ==
<references />

[[Category:How-To]]
[[Category:Deployment]]