You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Phabricator/Deployment: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Brennen Bearnes
imported>Brennen Bearnes
(Remove draft notice.)
 
(One intermediate revision by one other user not shown)
Line 1: Line 1:
{{draft}}
== Deployment windows ==
== Deployment windows ==


Line 48: Line 46:
** Always a lot of errors in here because of repo syncing, etc.
** Always a lot of errors in here because of repo syncing, etc.
* Check <code>/var/log/apache2/phabricator.log</code>
* Check <code>/var/log/apache2/phabricator.log</code>
== Downtime the hosts ==
If you can run cookbooks:
<pre>
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' phab1001
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' --force phabricator.wikimedia.org
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' --force phab.wmfusercontent.org
</pre>
If you can't run cookbooks:
=== Downtime Icinga for ===
* [https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=phab.wmfusercontent.org phab.wmfusercontent.org]
* [https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=phab1001 phab1001]
* [https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=phabricator.wikimedia.org phabricator.wikimedia.org]
=== Silence Alerts ===
# Click the bell on https://alerts.wikimedia.org/
# Click "New Silence"
# Under "Label" select "Instance" and toggle "regex" and add a new silence for each of the following:
:* <code>^(phab1001)(:[0-9]+)?$</code>
:* <code>^(phab\.wmfusercontent\.org)(:[0-9]+)?$</code>
:* <code>^(phabricator\.wikimedia\.org)(:[0-9]+)?$</code>
# Ensure you've put something in the comment field, "phab deployment" or whatever you're doing.


== Make milestone ==
== Make milestone ==

Latest revision as of 20:25, 18 July 2022

Deployment windows

Except for emergencies, Phabricator deployments should be conducted during mw:Phabricator/Maintenance windows.

Initial setup

You'll need:

Tagging a release

On your development system

  • Scripts for tagging and pushing
  • ./scripts/tag-release.sh
    • Tags all submodules
    • Outputs markup for milestone (but this sometimes fails)
    • Makes sure everything got pushed

For celerity updates:

On phab1001

  • /srv/phab is a clone of the deployment repo
  • Make sure it can pull
  • Fetch all submodules on prod
  • Ends at detached HEAD state
  • Make sure tags are there
  • update.sh in mukunda's homedir on phab1001
    • puppet disable
    • stops phd
    • checks out the version
    • stops apache and php
    • /usr/local/sbin/phab_deploy_finalize does db migrations and stuff
  • In #wikimedia-operations connect: !log starting phabricator deploy, momentary downtime expected while Apache restarts
  • sudo ~/update.sh and hope for best
  • systemctl status phd to make sure that's running
  • tail /var/log/phd/daemon.log to check for problems
    • Always a lot of errors in here because of repo syncing, etc.
  • Check /var/log/apache2/phabricator.log

Downtime the hosts

If you can run cookbooks:

sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' phab1001
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' --force phabricator.wikimedia.org
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' --force phab.wmfusercontent.org

If you can't run cookbooks:

Downtime Icinga for

Silence Alerts

  1. Click the bell on https://alerts.wikimedia.org/
  2. Click "New Silence"
  3. Under "Label" select "Instance" and toggle "regex" and add a new silence for each of the following:
  • ^(phab1001)(:[0-9]+)?$
  • ^(phab\.wmfusercontent\.org)(:[0-9]+)?$
  • ^(phabricator\.wikimedia\.org)(:[0-9]+)?$
  1. Ensure you've put something in the comment field, "phab deployment" or whatever you're doing.

Make milestone

  • Go into phab project and rename "next" milestone to the date of the deploy
    • i.e. 2020-02-13
    • paste generated markup with stuff from release
    • create a new "next" milestone
      • "This is for tasks ... next phabricator update"
    • mukunda's script used to do this automatically, may need some updating
    • archive the old one