You are browsing a read-only backup copy of Wikitech. The live site can be found at


From Wikitech-static
< Incidents
Revision as of 17:45, 8 April 2022 by imported>Krinkle (Krinkle moved page Incident documentation/20160319-Ores to Incidents/20160319-Ores)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search


ORES went down and responded slowly for ~2 hours today.


  • 1930 UTC: New deployment begins
  • 2005 UTC: ORES begins to be overloaded
  • 2025 UTC: A problem with old Jessie installs is discovered Phab:T130463 -- it turns out that it was really a pip issue with versioning
  • 2130 UTC: A new cluster is built and requests are being served at the rate that they come in
  • 2300 UTC: A new cluster configuration is complete.


  1. Pip does not remove old versions when installing new wheels. This will need to be done manually
  2. Our precaching utility will back-up during a short outage and unleash a load of requests on the service when it comes back online