You are browsing a read-only backup copy of Wikitech. The primary site can be found at

Incidents/2017-06-16 PHAB

From Wikitech-static
Jump to navigation Jump to search


Phabricator was down for approximately 8 minutes due to an unintentional/badly executed upgrade.


  • At approximately 13:18 UTC, I (User:20after4) attempted to fix a minor bug in a phabricator extension ( by deploying
  • Instead of cherry-picking the dependent change, the code was mistakenly fast-forwarded to a version which depended on new database migrations.
  • 13:19 UTC - icinga alerts and several people notice that phabricator is displaying a setup error page instead of the normal application.
  • 13:20 UTC - I !log in #wikimedia-operations to let people know that I'm working on it.
  • Because phabricator makes it easier to roll forward than to roll back, and my experience that migrations are usually fast, I ran the storage upgrade script to apply the migrations.
  • 13:22 UTC - I acknowledged the icinga alert to prevent further paging.
  • The script (phabricator/resources/sql/autopatches/20170528.maniphestdupes.php) was moderately large and slow compared to most phabricator migration scripts so this took several minutes to complete
  • 13:25 UTC - Service restored.


  • Almost every update to Phabricator requires database migrations to be applied. These are usually quite fast but not always.


  • Don't attempt code changes on friday, even trivial ones.
  • Always assume that code will have unexpected dependencies.
  • All upgrades, even minor ones should be during the scheduled window.
  • jynus requested that I coordinate with him to schedule the database slave to be offline when running migrations.