You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Monitoring/Latency

From Wikitech-static
< Monitoring
Revision as of 13:50, 28 June 2021 by imported>Wolfgang Kandek (Created page with "Icinga has alerts for increased latency at the Mediawiki appserver level. In June 2021 we had a case where several appservers entered a weird state that caused latency alerts...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Icinga has alerts for increased latency at the Mediawiki appserver level.

In June 2021 we had a case where several appservers entered a weird state that caused latency alerts. The solution was to restart PHP on the affected appservers. The following Prometheus query was used to find the appservers that were in the weird state:

(phpfpm_statustext_processes{cluster="appserver",state="idle"}) < 10

Then a

sudo restart-php7.2-fpm

This reoccurred 3 times in 2 days, we ended up restarting all appservers. Phabricator task: https://phabricator.wikimedia.org/T285634