You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Incident documentation/20160924-ORES: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Ladsgroup
(Created page with " == Summary == ORES review tool (= ORES extension) couldn't score edits made in 14 hours between rolling out of wmf.20 and the fast fix made in 2016-09-2...")
 
imported>Krinkle
 
Line 1: Line 1:
 
#REDIRECT [[Incidents/20160924-ORES]]
== Summary ==
ORES review tool (= [[mw:Extension:ORES|ORES extension]]) couldn't score edits made in 14 hours between rolling out of wmf.20 and the fast fix made in 2016-09-23
 
== Timeline ==
''This is a step by step outline of what happened to cause the incident and how it was remedied.''
* (2016-09-22) SAL: 20:00 thcipriani: rolling out wmf.20 to all wikis
* (2016-09-23) 9:44 The [[phab:T146461|phab task is created]]
* 9:45 The [https://gerrit.wikimedia.org/r/#/c/312491/ gerrit patch] is made to fix it in master
* 9:47 The patch is merged.
* 9:48 The [https://gerrit.wikimedia.org/r/#/c/312493/ backport to wmf.20] is made.
* 9:51 The backport is merged
* SAL: 09:58 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.20/extensions/ORES/includes/Cache.php: No int typehinting (causes jobs to crash) T146461 (duration: 00m 42s)
* SAL: 10:00 Amir1: ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki
* SAL: 10:05 Amir1: ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=wikidatawiki (T146461) and for 'trwiki', 'plwiki', 'fawiki', 'nlwiki', 'ruwiki', 'ptwiki'
== Conclusions ==
* There should be an alarm to scream when jobs such as ORESFetchScoreJob is not triggered for more than an hour.
* The lapse was easy to notice, ORES extension should have extensive CI tests.
 
== Actionables ==
<onlyinclude>
* Extensive CI tests for ORES extension ({{Phabricator|T146560}})
* High failure rate of account creation should trigger an alarm / page people ({{Phabricator|T146090}})
</onlyinclude>
 
[[Category:Incident documentation]]

Latest revision as of 17:45, 8 April 2022