You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Incident documentation/2018-02-20 thumbor

From Wikitech-static
< Incident documentation
Revision as of 21:30, 31 March 2021 by imported>Krinkle (Krinkle moved page Incident documentation/20180220-thumbor to Incident documentation/2018-02-20 thumbor)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Thumbor was unavailable for a brief period of time following a configuration deployment.

Summary

Thumbor, the image scaling service, became unavailable following a configuration deployment.

Timeline

  • 14:29 Filippo merges https://gerrit.wikimedia.org/r/407608
  • 14:44 The first errors of puppet failed on thumbor machines start coming in.
  • 14:48 thumbor.svc.eqiad.wmnet pages
  • 14:49 thumbor.svc.eqiad.wmnet recovers
  • 14:51 Revert of https://gerrit.wikimedia.org/r/407608 is merged on the puppet masters
  • 14:56 thumbor.svc.codfw.wmnet pages
  • 14:59 thumbor.svc.codfw.wmnet recovers

Conclusions

The puppet module for thumbor is configured to restart the thumbor instances (via thumbor-instances systemd unit, a meta-unit used to do mass actions on all thumbor instances configured on the host) upon config change. puppet issued a 'stop' to 'thumbor-instances' and subsequently a 'start'. Upon receiving a 'stop' all thumbor instances exited with status code 1 and considered in 'failed' state by systemd. The subsequent 'start' to 'thumbor-instances' failed to start the instances properly.

Recovery of thumbor happened on the next puppet run when the "ensure" mechanism of puppet made sure all thumbor instances configured were running as expected. During the outage there has been also a fair amount of monitoring spam due to individual alerts for each thumbor instance in 'failed' state.

During the incident, about 0.08% of requests for thumbnails failed and end users experienced the 5xx emitted during unavailability for about 10 minutes.

20180220-thumbor-availability.png

Actionables

  • Don't issue uncoordinated restarts of thumbor from puppet: gerrit:412934
  • Eliminate per thumbor-instance icinga checks and related spam: gerrit:412948