You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Phabricator/Deployment: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Brennen Bearnes
(→‎Initial setup: Link dev env notes)
 
imported>Brennen Bearnes
(Remove draft notice.)
 
(3 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{draft}}
== Deployment windows ==
== Deployment windows ==


Line 9: Line 7:
You'll need:
You'll need:


* A checkout of https://gerrit.wikimedia.org/r/plugins/gitiles/phabricator/deployment/ and all submodules.
* A checkout of https://gerrit.wikimedia.org/g/phabricator/deployment/ and all submodules.
* A local install of PHP 7.3.
* A local install of PHP 7.3.
* A running Phabricator install
* A running Phabricator install
Line 18: Line 16:
=== On your development system ===
=== On your development system ===


* scripts for tagging and pushing
* Scripts for tagging and pushing
* ./scripts/tag-release.sh
* <code>./scripts/tag-release.sh</code>
** tags all submodules
** Tags all submodules
** outputs markup for milestone (but this sometimes fails)
** Outputs markup for milestone (but this sometimes fails)
** make sure everything got pushed
** Makes sure everything got pushed
 
For celerity updates:
 
* https://secure.phabricator.com/book/phabcontrib/article/adding_new_css_and_js/
** The dependency and packaging maps are generated by bin/celerity map, which updates resources/celerity/map.php.


=== On phab1001 ===
=== On phab1001 ===


* /srv/phab
* /srv/phab is a clone of the deployment repo
* make sure it can pull
* Make sure it can pull
* fetch all submodules on prod
* Fetch all submodules on prod
* ends at detached HEAD state
* Ends at detached HEAD state
* make sure tags are there
* Make sure tags are there
* <code>update.sh</code> in mukunda's homedir on phab1001
* <code>update.sh</code> in mukunda's homedir on phab1001
** puppet disable
** puppet disable
Line 36: Line 39:
** checks out the version
** checks out the version
** stops apache and php
** stops apache and php
** /usr/local/sbin/phab_deploy_finalize does db migrations and stuff
** <code>/usr/local/sbin/phab_deploy_finalize</code> does db migrations and stuff
* In {{irc|wikimedia-operations}}: <code>!log starting phabricator deploy, momentary downtime expected while apache restarts</code>
* In {{irc|wikimedia-operations}}: <code>!log starting phabricator deploy, momentary downtime expected while Apache restarts</code>
* <code>sudo ~/update.sh</code> and hope for best
* <code>sudo ~/update.sh</code> and hope for best
* <code>systemctl status phd</code> to make sure that's running
* <code>systemctl status phd</code> to make sure that's running
* <code>tail /var/log/phd.log</code> to check for problems
* <code>tail /var/log/phd/daemon.log</code> to check for problems
** always a lot of errors in here because of repo syncing, etc.
** Always a lot of errors in here because of repo syncing, etc.
* check /var/log/apache2/phabricator.log
* Check <code>/var/log/apache2/phabricator.log</code>
 
== Downtime the hosts ==
 
If you can run cookbooks:
<pre>
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' phab1001
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' --force phabricator.wikimedia.org
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' --force phab.wmfusercontent.org
</pre>
 
If you can't run cookbooks:
 
=== Downtime Icinga for ===
* [https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=phab.wmfusercontent.org phab.wmfusercontent.org]
* [https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=phab1001 phab1001]
* [https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=phabricator.wikimedia.org phabricator.wikimedia.org]
 
=== Silence Alerts ===
# Click the bell on https://alerts.wikimedia.org/
# Click "New Silence"
# Under "Label" select "Instance" and toggle "regex" and add a new silence for each of the following:
:* <code>^(phab1001)(:[0-9]+)?$</code>
:* <code>^(phab\.wmfusercontent\.org)(:[0-9]+)?$</code>
:* <code>^(phabricator\.wikimedia\.org)(:[0-9]+)?$</code>
# Ensure you've put something in the comment field, "phab deployment" or whatever you're doing.


== Make milestone ==
== Make milestone ==

Latest revision as of 20:25, 18 July 2022

Deployment windows

Except for emergencies, Phabricator deployments should be conducted during mw:Phabricator/Maintenance windows.

Initial setup

You'll need:

Tagging a release

On your development system

  • Scripts for tagging and pushing
  • ./scripts/tag-release.sh
    • Tags all submodules
    • Outputs markup for milestone (but this sometimes fails)
    • Makes sure everything got pushed

For celerity updates:

On phab1001

  • /srv/phab is a clone of the deployment repo
  • Make sure it can pull
  • Fetch all submodules on prod
  • Ends at detached HEAD state
  • Make sure tags are there
  • update.sh in mukunda's homedir on phab1001
    • puppet disable
    • stops phd
    • checks out the version
    • stops apache and php
    • /usr/local/sbin/phab_deploy_finalize does db migrations and stuff
  • In #wikimedia-operations connect: !log starting phabricator deploy, momentary downtime expected while Apache restarts
  • sudo ~/update.sh and hope for best
  • systemctl status phd to make sure that's running
  • tail /var/log/phd/daemon.log to check for problems
    • Always a lot of errors in here because of repo syncing, etc.
  • Check /var/log/apache2/phabricator.log

Downtime the hosts

If you can run cookbooks:

sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' phab1001
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' --force phabricator.wikimedia.org
sudo cookbook sre.hosts.downtime -H 1 -r 'bug fix' --force phab.wmfusercontent.org

If you can't run cookbooks:

Downtime Icinga for

Silence Alerts

  1. Click the bell on https://alerts.wikimedia.org/
  2. Click "New Silence"
  3. Under "Label" select "Instance" and toggle "regex" and add a new silence for each of the following:
  • ^(phab1001)(:[0-9]+)?$
  • ^(phab\.wmfusercontent\.org)(:[0-9]+)?$
  • ^(phabricator\.wikimedia\.org)(:[0-9]+)?$
  1. Ensure you've put something in the comment field, "phab deployment" or whatever you're doing.

Make milestone

  • Go into phab project and rename "next" milestone to the date of the deploy
    • i.e. 2020-02-13
    • paste generated markup with stuff from release
    • create a new "next" milestone
      • "This is for tasks ... next phabricator update"
    • mukunda's script used to do this automatically, may need some updating
    • archive the old one