You are browsing a read-only backup copy of Wikitech. The live site can be found at

Incident documentation/20161202-20161201-PageImages: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
(One intermediate revision by one other user not shown)
Line 1: Line 1:
== Summary ==
#REDIRECT [[Incidents/20161202-20161201-PageImages]]
A change went live on the train that was supposed to update the page images API to provide an additional api parameter that would allow you to select a page image depending on license type. The current behaviour was that the page images API would return images that were free. The new parameter would allow API callers to also explicitly include images that were non-free. Due to an issue in implementation, the new change reset all known free page images and as a result for most API requests the page images API would return no image. This caused images that previously were appearing in many of Wikimedias projects including our Wikipedia apps, the Wikimedia portal and mobile web search results to disappear for about 3 hours.
== Timeline ==
* Dan Garry commented that images had disappeared from the portal. Asked @ebernhardson to drop everything and fix it.
* @MaxSem tracked down the [ offending change]
* @ebernhardson pinged reading web team (Baha and Jon R) to make them aware of his proposed fix
* @jdlrobson and @ebernhardson chatted and agreed [ patch to enable `any` image as default] was best short term solution and patch was merged
* @eberndardson  SWATed the fix
* Discovery team enabled a script to re-populate all images on the script. It is estimated that it will take 3 weeks to run. In the meantime API subscribers will be returned any image by default in all Wikimedia projects (which may include non-free images). If they request free images they will only get free images, but there will be many missing images until the maintenance script has completed. Requests for an image with license type `any` may also return an inferior result compared to what images are available.
== Conclusions ==
* Had this change been enabled explicitly by a config changes it would have minimised the time frame from this change going live and being detected and minimised disruption. We should consider flagging big changes like this in future.
* Changes to the PageImages extension should probably be run via the discovery team in future and have been subjected to more thorough review.
* These kinds of big changes should have been thoroughly tested on beta cluster before being enabled in production servers.
== Actionables ==
* [ Update documentation of extension to make developers wary of this problem]
* [[phab:T152252|Improve maintenance script so that recovery from such problems can be quicker.]]
[[Category:Incident documentation]]

Latest revision as of 17:45, 8 April 2022