You are browsing a read-only backup copy of Wikitech. The live site can be found at


From Wikitech-static
Jump to navigation Jump to search


For about 12 hours 70% of event logging events did not include an schema field and thus did not validate

Issue affected client side events, which are the majority of events sent. Server side events and events sent via client apps were unaffected.

File:EventLogging issue January 7th 2015.png
EventLogging issue January 7th 2015


  • Changeset [1] gets merged on January 6th
  • Changeset gets deployed at 2 am UTC on January 7th to mediawiki
  • Alarms are raised on throughput of events
  • Look at logs, establish cause and send patchset to fix it
  • Fix patch is deployed on the UTC evening (morning on the west coast)


Deployments should be tested on Beta Cluster, we should avoid going directly to production


We actually learned about this issue via the alarms so, while we could have alarms on the database, which we do not, the alarms on throughput did their job