You are browsing a read-only backup copy of Wikitech. The live site can be found at

Incident documentation/20130617-AccountCreation

From Wikitech-static
Jump to navigation Jump to search


Due to code in VisualEditor's A/B testing sorting on new accounts, newly created user accounts were not having their email and password set. Thus, many thousands of new accounts were only partially created.

Reported as bugzilla:49727.

What Happened

Basically, this commit 68845 caused the whole thing. This one 69277 fixed it.

Lessons Learned

Our logging works, but we don't monitor those logs and report spikes/errors.

Action Items

  • Errors and exceptions are currently broadcast to fluorine and vanadium via UDP. Ori has code that parses the stream and generates the Ganglia graphs, but it isn't hooked up to Icinga or any other form of monitoring. We need someone from Ops to pair with Ori to set this up. This is bugzilla:49757. Yes Done