You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Incident documentation/20160713-ContentTranslation: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Nikerabbit
imported>Krinkle
 
Line 1: Line 1:
== Summary ==
#REDIRECT [[Incidents/20160713-ContentTranslation]]
ContentTranslation extension used up all available SQL connections causing issues for extensions using the shared database (at least Flow, Echo). This was caused up by thousands of hits in short time to the translation draft saving API by on user. What caused those requests is still unknown.
 
== Timeline ==
See also {{diff|740361|oldid=739725|label=Server admin log entries during this time}}.
 
* 2016-07-12 [15:48:53] (UTC) icinga-wm> PROBLEM - MariaDB Slave SQL: x1 on db1031 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
* [15:50] jynus notices x1 master is down
* [15:52] jynus identified ContentTranslation\Translation::update / cx_translations as the cause
* [15:59] https://phabricator.wikimedia.org/T140123 is created by jynus
* [16:05] Language team developers were pinged in IRC
* [16:10] Nikerabbit notices the ping and starts checking what is going on
* [16:12] Patch to disable CX is deployed
* [16:19] Nikerabbit checked the code and believes it can only be caused by high number of API hits (no loops in the relevant code etc.)
* [16:28] Nikerabbit is able to confirm this is true via https://graphite.wikimedia.org/render/?width=586&height=308&target=MediaWiki.api.cxsave.executeTiming.count [https://phabricator.wikimedia.org/T140123#2454099 static copy]. Based on other data we assume these are all coming from one or at most few users.
* [16:30-16:35] Brainstorming for solutions: blocking the user, poolcounter, ping limiter
* [16:40] Nikerabbit thinks ping limiter is the best option and starts working on the patches, ostriches indicates he can help with review and deploy
* [16:54] Nikerabbit has finished the patches: https://gerrit.wikimedia.org/r/#/q/topic:cxsave,n,z
* [16:55-17:45] Patches are being reviewed and deployed. Tests for CX extension patch takes 10+ minutes and they need to run multiple times.
* [17:48-18:08] Patch to turn CX back on is created and deployed
* [18:09-18:13] Initially CX does not work. Special pages only display user toolbar, no JavaScript errors. After few minutes it starts to work, indicating it was caused by the 5 min caching.
 
== Conclusions ==
* Surprised there is no general request limiting automatically applied, but code changes were required.
* Language team has no monitoring in place that would have detected this issue.
 
== Actionables ==
<onlyinclude>
* {{Status|done}} The ping limiter which was added can stay in place. The values should be verified. [[Phab:T140615|T140615]]
* {{Status|done}} Audit the front-end code for possible causes of these requests. [[Phab:T140622|T140622]]
* {{Status|done}} Try to contact the user for more information and advice [[Phab:T140619|T140619]]
</onlyinclude>
[[Category:Incident documentation]]

Latest revision as of 17:45, 8 April 2022

Redirect to:

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Incident_documentation/20160713-ContentTranslation&oldid=589033"