You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Heterogeneous deployment/Train deploys: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Chad
(→‎Remove symlinks to expired branches: static isn't tracked there anymore and doesn't need manual rm.)
imported>Thcipriani
(→‎Update roadmap: 1.28 roadmap)
Line 31: Line 31:
* <code>./make-wmf-branch -n VERSION -o master</code>
* <code>./make-wmf-branch -n VERSION -o master</code>
** (e.g. <code>php make-wmf-branch -n 1.27.0-wmf.9 -o master</code>)
** (e.g. <code>php make-wmf-branch -n 1.27.0-wmf.9 -o master</code>)
=== Remove symlinks to expired branches ===
Any branch checkout on the deployment server that has not been used for more than 5 weeks can be safely removed to reduce disk usage across the cluster.
* <code>/srv/mediawiki-staging/multiversion/deleteMediaWiki STALE_VERSION</code>
* <code>DOLOGMSGNOLOG=1 git commit -m "Remove STALE_VERSION symlinks"</code>


=== Clone new branch ===
=== Clone new branch ===
Line 49: Line 43:
** You can check a core patch to see if it will apply cleanly with <code>git apply --check --3way /srv/patches/VERSION/core/XX-TXXX.patch</code>
** You can check a core patch to see if it will apply cleanly with <code>git apply --check --3way /srv/patches/VERSION/core/XX-TXXX.patch</code>
** If the patch checks out, apply and commit it with <code>git am --3way /srv/patches/VERSION/core/XX-TXXX.patch</code>
** If the patch checks out, apply and commit it with <code>git am --3way /srv/patches/VERSION/core/XX-TXXX.patch</code>
** If the patch fails to apply, investigate whether it's due to a conflict (<code>git status</code>) or the patch having been merged since the new branch cut (search <code>git log</code> for the commit, etc.). If it turns out to be the latter, remove the patch file from the <code>/srv/patches/VERSION</code> directory. If you need extra help, contact {{ircnick|ostriches|Chad}} or {{ircnick|csteipp|Chris}} in IRC ({{irc|mediawiki_security}}).
** If the patch fails to apply, investigate whether it's due to a conflict (<code>git status</code>) or the patch having been merged since the new branch cut (search <code>git log</code> for the commit, etc.). If it turns out to be the latter, remove the patch file from the <code>/srv/patches/VERSION</code> directory. If you need extra help, contact {{ircnick|ostriches|Chad}} or {{ircnick|dapatrick|Darian}} in IRC ({{irc|mediawiki_security}}).


=== Create patches to update wikiversions.json ===
=== Create patches to update wikiversions.json ===
Line 67: Line 61:
* <code>cd /srv/mediawiki-staging</code>
* <code>cd /srv/mediawiki-staging</code>
* <code>DOLOGMSGNOLOG=1 git reset --hard origin/master</code>
* <code>DOLOGMSGNOLOG=1 git reset --hard origin/master</code>
=== Restore symlinks on deployment server ===
* Review and submit symlinks patches in gerrit
** This includes the patches for any expired branches that were removed and the new branch that was added.
* Wait for gerrit/zuul/jenkins to merge the patch(es)
* Pull patch(es) to deployment server
** <code>cd /srv/mediawiki-staging</code>
** <code>git fetch</code>
* Check diff to ensure it is what you expect
** <code>git diff HEAD..origin/master</code>
* Apply changes
** <code>git rebase origin/master</code>


=== Remove clones of expired branches ===
=== Remove clones of expired branches ===
For each branch that you [[#Remove_symlinks_to_expired_branches|removed symlinks for]], remove the clone from <code>/srv/mediawiki-staging</code>:
For each branch that is more than 5 versions old, remove the clone from <code>/srv/mediawiki-staging</code>:
* <code>rm -rf /srv/mediawiki-staging/php-STALE_VERSION</code>
* <code>rm -rf /srv/mediawiki-staging/php-STALE_VERSION</code>


Line 88: Line 70:
** Do not commit and push to gerrit, only make this change locally on the deployment seerver
** Do not commit and push to gerrit, only make this change locally on the deployment seerver
* Run [[scap]] to (re)build localization caches and sync changes across the cluster
* Run [[scap]] to (re)build localization caches and sync changes across the cluster
** <code>scap "testwiki to php-VERSION and rebuild l10n cache"</code>
** <code>scap sync "testwiki to php-VERSION and rebuild l10n cache"</code>
* Verify version change and l10n cache on [https://test.wikipedia.org/wiki/Special:Version testwiki]
* Verify version change and l10n cache on [https://test.wikipedia.org/wiki/Special:Version testwiki]
* Revert local changes: <code>git checkout -- wikiversions.json</code>
* Revert local changes: <code>git checkout -- wikiversions.json</code>
Line 94: Line 76:
=== Remove left over files from expired branches ===
=== Remove left over files from expired branches ===
Because we don't rsync the localization CDB files during [[scap]], some files will be left behind after scap runs for expired branches that were removed. Clean them up across the cluster using <code>dsh</code>
Because we don't rsync the localization CDB files during [[scap]], some files will be left behind after scap runs for expired branches that were removed. Clean them up across the cluster using <code>dsh</code>
* <code>SSH_AUTH_SOCK=/run/keyholder/proxy.sock dsh -F 20 -M -g mediawiki-installation -r ssh -o -oUser=mwdeploy -- rm -rf /srv/mediawiki/php-STALE_VERSION</code>
* <code>SSH_AUTH_SOCK=/run/keyholder/proxy.sock dsh -F 20 -M -g scap-masters -r ssh -o -oUser=mwdeploy -- rm -rf /srv/mediawiki/php-[VERSION]</code>
* <code>SSH_AUTH_SOCK=/run/keyholder/proxy.sock dsh -F 20 -M -g scap-proxies -r ssh -o -oUser=mwdeploy -- rm -rf /srv/mediawiki/php-[VERSION]</code>
* <code>SSH_AUTH_SOCK=/run/keyholder/proxy.sock dsh -F 20 -M -g mediawiki-installation -r ssh -o -oUser=mwdeploy -- rm -rf /srv/mediawiki/php-[VERSION]</code>


=== Wait for deploy window ===
=== Wait for deploy window ===
Line 123: Line 107:
* Clone mediawiki/tools/release.git
* Clone mediawiki/tools/release.git
** <code>git clone ssh://gerrit.wikimedia.org:29418/mediawiki/tools/release.git</code>
** <code>git clone ssh://gerrit.wikimedia.org:29418/mediawiki/tools/release.git</code>
* Add wiki credentials to make-deploy-notes/uploadChangelog.php
* Copy <code>make-deploy-notes/auth.php.example</code> to <code>make-deploy-notes/auth.php</code> and fill in your mediawiki.org useraname and password.
** <code>$wiki->login( 'my user account', 'my awesome password' );</code>
** <code>$wiki->login( 'my user account', 'my awesome password' );</code>
* Clone mediawiki/core
* Clone mediawiki/core
Line 143: Line 127:


=== Update roadmap ===
=== Update roadmap ===
* Change the ''Deployed to group'' (if you're using VisualEditor) or the 3rd parameter of the ''WMFReleaseTableRow'' template (if you're using the wikitext editor) to "0" (deployed to group0)n [[:mw:MediaWiki 1.27/Roadmap|the branch roadmap]].
* Change the ''Deployed to group'' (if you're using VisualEditor) or the 3rd parameter of the ''WMFReleaseTableRow'' template (if you're using the wikitext editor) to "0" (deployed to group0)n [[:mw:MediaWiki 1.28/Roadmap|the branch roadmap]].


== Wednesday: group0 to group1 deploy ==
== Wednesday: group0 to group1 deploy ==

Revision as of 22:27, 31 May 2016

Train wreck at Montparnasse 1895.jpg

Tuesday: New branch creation and deploy

Create the new branch in gerrit

The new branch can be created in Gerrit from anywhere. It is often faster to do this step on a host in the cluster to minimize the time needed to clone from gerrit.

The script is run as your regular user member of the wikidev group (as of Feb 16th 2016). You have to configure git:

  • git config --global user.name "FIRST NAME LAST NAME"
  • git config --global user.email "youremail@example.org"

Use ssh instead of https for pushes:

  • git config --global url.'ssh://gerrit.wikimedia.org:29418'.pushInsteadOf "https://gerrit.wikimedia.org/r/p"

Create a a ssh key pair and load it:

  • ssh-keygen -f ~/.ssh/makebranch && cat ~/.ssh/makebranch.pub
  • ssh-agent bash && ssh-add ~/.ssh/makebranch

Upload the public key (~/.ssh/makebranch.pub) to your user account in Gerrit: https://gerrit.wikimedia.org/r/#/settings/ssh-keys


Tweak ssh config to use your Gerrit username and the ssh key you have generated:

Host gerrit.wikimedia.org
        IdentityFile ~/.ssh/makebranch
        IdentitiesOnly yes
        User  LABS_ACCOUNT_NAME
  • git clone ssh://gerrit.wikimedia.org:29418/mediawiki/tools/release.git
  • cd release/make-wmf-branch
  • ./make-wmf-branch -n VERSION -o master
    • (e.g. php make-wmf-branch -n 1.27.0-wmf.9 -o master)

Clone new branch

  • On tin, to create a new /srv/mediawiki-staging/php-VERSION directory run:
    • /srv/mediawiki-staging/multiversion/checkoutMediaWiki VERSION php-VERSION
    • (e.g. /srv/mediawiki-staging/multiversion/checkoutMediaWiki 1.27.0-wmf.9 php-1.27.0-wmf.9)

Apply security patches

  • Look for a patches directory at /srv/patches/VERSION. If one doesn't yet exist, copy over the previous version's directory. (e.g., cp -r /srv/patches/1.27.0-wmf.9 /srv/patches/1.27.0-wmf.10)
  • Patches should be named sequentially in the order that they will cleanly apply (e.g. 01-T123.patch, 02-T321.patch)
  • Check and apply each patch in both /srv/patches/VERSION/core and /srv/patches/VERSION/extensions/NAME to the new core checkout and extensions, respectively.
    • You can check a core patch to see if it will apply cleanly with git apply --check --3way /srv/patches/VERSION/core/XX-TXXX.patch
    • If the patch checks out, apply and commit it with git am --3way /srv/patches/VERSION/core/XX-TXXX.patch
    • If the patch fails to apply, investigate whether it's due to a conflict (git status) or the patch having been merged since the new branch cut (search git log for the commit, etc.). If it turns out to be the latter, remove the patch file from the /srv/patches/VERSION directory. If you need extra help, contact Chad (ostriches) or Darian (dapatrick) in IRC (#mediawiki_security connect).

Create patches to update wikiversions.json

  • Create group0 to VERSION patch
    • /srv/mediawiki-staging/multiversion/updateWikiversions group0 php-VERSION
      • (e.g. /srv/mediawiki-staging/multiversion/updateWikiversions group0 php-1.27.0-wmf.9)
    • git add wikiversions.json
    • DOLOGMSGNOLOG=1 git commit -m "Group0 to VERSION"
      • (e.g. DOLOGMSGNOLOG=1 git commit -m "Group0 to 1.27.0-wmf.9")

Send staged patches to gerrit for review

  • cd /srv/mediawiki-staging
  • git push origin HEAD:refs/for/master/VERSION
    • (e.g. git push origin HEAD:refs/for/master/1.27.0-wmf.9)

Discard changes to working directory and index on tin

  • cd /srv/mediawiki-staging
  • DOLOGMSGNOLOG=1 git reset --hard origin/master

Remove clones of expired branches

For each branch that is more than 5 versions old, remove the clone from /srv/mediawiki-staging:

  • rm -rf /srv/mediawiki-staging/php-STALE_VERSION

Sync to cluster and verify on testwiki

  • Edit /srv/mediawiki-staging/wikiversions.json and set testwiki to php-VERSION
    • Do not commit and push to gerrit, only make this change locally on the deployment seerver
  • Run scap to (re)build localization caches and sync changes across the cluster
    • scap sync "testwiki to php-VERSION and rebuild l10n cache"
  • Verify version change and l10n cache on testwiki
  • Revert local changes: git checkout -- wikiversions.json

Remove left over files from expired branches

Because we don't rsync the localization CDB files during scap, some files will be left behind after scap runs for expired branches that were removed. Clean them up across the cluster using dsh

  • SSH_AUTH_SOCK=/run/keyholder/proxy.sock dsh -F 20 -M -g scap-masters -r ssh -o -oUser=mwdeploy -- rm -rf /srv/mediawiki/php-[VERSION]
  • SSH_AUTH_SOCK=/run/keyholder/proxy.sock dsh -F 20 -M -g scap-proxies -r ssh -o -oUser=mwdeploy -- rm -rf /srv/mediawiki/php-[VERSION]
  • SSH_AUTH_SOCK=/run/keyholder/proxy.sock dsh -F 20 -M -g mediawiki-installation -r ssh -o -oUser=mwdeploy -- rm -rf /srv/mediawiki/php-[VERSION]

Wait for deploy window

All of the changes above can be done at any time prior to the actual deployment window. Depending on how practiced you are and where you choose to run commands (full clones of mediawiki-core from outside the cluster can take a while) the steps up to this point will typically take 60 to 120 minutes.

Switch group0 wikis to VERSION

  • Review and submit group0 to VERSION patch in gerrit
  • Wait for gerrit/zuul/jenkins to merge the patch(es)
  • Pull patch(es) to deployment server
    • cd /srv/mediawiki-staging
    • git fetch
  • Check diff to ensure it is what you expect
    • git diff HEAD..origin/master
  • Apply changes
    • git rebase origin/master
  • Sync the change across the cluster
    • sync-wikiversions "group0 to VERSION"
      • (e.g. sync-wikiversions "group0 to 1.24wmf4")
  • Verify that mediawikiwiki switched to the new version
  • Monitor irc and logstash and/or fatalmonitor for problems

Purge localization cache for now unused versions

  • scap-purge-l10n-cache --version VERSION-2
    • (e.g. scap-purge-l10n-cache --version 1.24wmf2)

Update deploy notes

Setup the tools you need to do this:

For the new branch and any branches that may have changed in the last week:

  • Check out the branch locally
    • git checkout wmf/VERSION
    • (e.g. git checkout wmf/1.24wmf4)
  • If you don't already have the previous branch checked out, do that as well
    • git checkout wmf/PREVIOUS-VERSION
    • (e.g. git checkout wmf/1.24wmf3)
  • Update the submodules
    • git submodule update --init --recursive
  • Run the change log generation and upload script
    • php path/to/make-deploy-notes/uploadChangelog.php wmf/VERSION
      • (e.g. php path/to/make-deploy-notes/uploadChangelog.php wmf/1.24wmf4)
  • Repeat for additional branches as needed

Update roadmap

  • Change the Deployed to group (if you're using VisualEditor) or the 3rd parameter of the WMFReleaseTableRow template (if you're using the wikitext editor) to "0" (deployed to group0)n the branch roadmap.

Wednesday: group0 to group1 deploy

  • Clone the MediaWiki release tools to your home directory on tin
  • Use the deploy-promote script inside ~/release/bin/ to update wikiversions.json
    • You will be prompted to verify the update, then you will be asked to approve relevant patches in gerrit
    • ~/release/bin/deploy-promote
      • Promote group1 from VERSION-1 to VERSION? Enter to continue, ctl-c to cancel:
    • Once you have +2'd the patches in gerrit, and they have merged into the config repo, hit enter at the 2nd prompt
      • Now merge the patch and press enter to continue with git pull && sync-wikiversions
    • After the script run is complete, group1 wikis should be running VERSION

Thursday: group{0,1} to all deploy

  • Thursday deploy is very similar to the Wednesday deploy, the only difference in terms of procedure is the target group
  • Clone the MediaWiki release tools to your home directory on tin
  • Use the deploy-promote script inside ~/release/bin/ to update wikiversions.json
    • You will be prompted to verify the update, then you will be asked to approve relevant patches in gerrit
    • ~/release/bin/deploy-promote all
      • Promote all from VERSION-1 to VERSION? Enter to continue, ctl-c to cancel:
    • Once you have +2'd the patches in gerrit, and they have merged into the config repo, hit enter at the 2nd prompt
      • Now merge the patch and press enter to continue with git pull && sync-wikiversions
    • After the script run is complete, all wikis should be running VERSION