Incident status

• cirrussearch restart

No incidents reported in the last 14 days.

Create a new incident report

1. Deal with the issue (See also Incident response).
2. Decide whether to write a lightweight or full-length report, and use the below form to start.
   Lightweight report (template): <inputbox>

 type=create
 editintro=Incident response/Report editintro
 preload=Incident response/Lightweight report template
 default=
 placeholder=Enter service name...
 buttonlabel=Create report
 bgcolor=inherit
 break=no
 prefix=Incident documentation/2021-09-26_

</inputbox>
Full-length report (template): <inputbox>

 type=create
 editintro=Incident response/Report editintro
 preload=Incident response/Full report template
 default=
 placeholder=Enter service name...
 buttonlabel=Create report
 bgcolor=inherit
 break=no
 prefix=Incident documentation/2021-09-26_

</inputbox>

1. Write up what happened, filling out the template. Use past incident reports for inspiration.
2. Once done, change the status to "in-review", preferably within 7 days of the incident.
   Within 3 weeks the ONFIRE working group will get back to you with feedback.
   After the ONFIRE review and any updates to the report, set the status to "final" and notify the ops@ mailing list.

Past incidents

This page tracks incident reports about Wikimedia Foundation properties, such as Wikipedia.

<inputbox> type=fulltext prefix=Incident documentation/ break=no width=30 searchbuttonlabel=Search placeholder=Enter search term... namespace= </inputbox>

2021

• 2021-01-16 appserver latency
• 2021-02-01 swift-codfw
• 2021-02-05 wmflabs certs expired
• 2021-02-26 sudo
• 2021-03-14 Mediawiki API
• 2021-03-30 Jobqueue overload
• 2021-04-03 mailman2 attachments
• 2021-04-06 partial rack codfw
• 2021-04-15 appserver latency
• 2021-04-20 MediaWiki API slowdown
• 2021-04-26 thumbor
• 2021-04-27 Commons wiki primary db
• 2021-04-29 db and memc load
• 2021-06-15 Eqsin network
• 2021-06-29 trwikivoyage primary db
• 2021-06-30 Ulsfo traffic surge
• 2021-07-14 eventgate-analytics latency spike caused MW app server overload
• 2021-07-16 asw-a2-codfw network
• 2021-07-26 ruwikinews DynamicPageList
• 2021-09-01 partial parsoid outage
• 2021-09-06 Wikifeeds
• 2021-09-13 cirrussearch restart
• Table of outages

Incident history

Check Category:Incident documentation for older incident reports.