You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Phabricator/Deployment

From Wikitech-static
< Phabricator
Revision as of 20:25, 18 July 2022 by imported>Brennen Bearnes (Remove draft notice.)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Deployment windows

Except for emergencies, Phabricator deployments should be conducted during mw:Phabricator/Maintenance windows.

Initial setup

You'll need:

Tagging a release

On your development system

  • Scripts for tagging and pushing
  • ./scripts/tag-release.sh
    • Tags all submodules
    • Outputs markup for milestone (but this sometimes fails)
    • Makes sure everything got pushed

For celerity updates:

On phab1001

  • /srv/phab is a clone of the deployment repo
  • Make sure it can pull
  • Fetch all submodules on prod
  • Ends at detached HEAD state
  • Make sure tags are there
  • update.sh in mukunda's homedir on phab1001
    • puppet disable
    • stops phd
    • checks out the version
    • stops apache and php
    • /usr/local/sbin/phab_deploy_finalize does db migrations and stuff
  • In #wikimedia-operations connect: !log starting phabricator deploy, momentary downtime expected while Apache restarts
  • sudo ~/update.sh and hope for best
  • systemctl status phd to make sure that's running
  • tail /var/log/phd/daemon.log to check for problems
    • Always a lot of errors in here because of repo syncing, etc.
  • Check /var/log/apache2/phabricator.log

Downtime the hosts

If you can run cookbooks:

sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' phab1001
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' --force phabricator.wikimedia.org
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' --force phab.wmfusercontent.org

If you can't run cookbooks:

Downtime Icinga for

Silence Alerts

  1. Click the bell on https://alerts.wikimedia.org/
  2. Click "New Silence"
  3. Under "Label" select "Instance" and toggle "regex" and add a new silence for each of the following:
  • ^(phab1001)(:[0-9]+)?$
  • ^(phab\.wmfusercontent\.org)(:[0-9]+)?$
  • ^(phabricator\.wikimedia\.org)(:[0-9]+)?$
  1. Ensure you've put something in the comment field, "phab deployment" or whatever you're doing.

Make milestone

  • Go into phab project and rename "next" milestone to the date of the deploy
    • i.e. 2020-02-13
    • paste generated markup with stuff from release
    • create a new "next" milestone
      • "This is for tasks ... next phabricator update"
    • mukunda's script used to do this automatically, may need some updating
    • archive the old one