Incident status

• Unintentional +2 on a config patch without deployment

• eqiad/LVS

• wdqs CODFW 5xx errors

• videoscaler/jobrunner

No incidents reported in the last 14 days.

Create a new incident report

1. Deal with the issue by following Incident response.
2. Use this form to create a report based on the report template:
   <inputbox>

 type=create
 useve=1
 preload=Incident response/Full report template
 default=2023-06-02 myservice
 placeholder=YYYY-MM-DD Service name here
 buttonlabel=Create report
 bgcolor=inherit
 break=no
 prefix=Incidents/

</inputbox>

1. Write up what happened, filling out the template. Use past incident reports for inspiration.

Past incidents

This page tracks incident reports about Wikimedia Foundation properties, such as Wikipedia.

<inputbox> type=fulltext prefix=Incidents/ break=no width=30 searchbuttonlabel=Search placeholder=Enter search term... namespace= </inputbox>

2023

• 2023-01-09 API errors - db1143
• 2023-01-10 eqsin network outage
• 2023-01-14 asw-b2-codfw failure
• 2023-01-17 MediaWiki
• 2023-01-17 asw-b2-codfw failure redux
• 2023-01-24 sessionstore quorum issues
• 2023-01-30 kartotherian
• 2023-02-07 mediawiki.page-undelete event stream
• 2023-02-11 logstash latency
• 2023-02-22 read only
• 2023-02-22 wiki outage
• 2023-02-23 PHP worker threads exhaustion
• 2023-02-28 GitLab data loss
• 2023-03-09 mailman
• 2023-04-17 eqiad/LVS
• 2023-05-02 Phabricator outage during GitLab maintenance
• 2023-05-05 prometheus down in ulsfo and eqsin
• 2023-05-05 prometheus ulsfo and eqsin
• 2023-05-05 wdqs not updating in codfw
• 2023-05-19 videoscaler/jobrunner
• 2023-05-23 wdqs CODFW 5xx errors
• 2023-05-25 eqiad/LVS
• 2023-05-30 Unintentional +2 on a config patch without deployment

Incident history

Check Category:Incident documentation for older incident reports.

See also

• https://www.wikimediastatus.net (docs).

• Incident graphs by Krinkle.

• #wikimedia-tech ^connect on Libera Chat