You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Wikidough/Monitoring

From Wikitech-static
< Wikidough
Revision as of 17:35, 25 April 2022 by imported>Sukhbir Singh (improve basic check resolving message)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Wikidough Basic Check

What does it mean?

If this check fails, it means that a Wikidough host is not responding on ports 443 and/or 853 on its IPv4 and/or IPv6 address. This happens when the host is down, depooled, or the dnsdist.service is inactive or has failed.

Resolving this message

  • Head to Icinga and check if the host is up and if there are other checks that are failing (which may indicate a problem with the host itself).
  • Is the dnsdist.service active? Check with: systemctl status dnsdist.service.
    • If it is has stopped or failed, try restarting it. If it fails again, check the journal output to see why it is failing.
  • Since this check queries the host IP and not the anycast IP, it is unlikely that there is an issue with anycast-healthchecker or the bird or bird6 services.
    • Nevertheless, checking the status of the above three services might be worthwhile.

Service Restart Check

What does it mean?

A failure of this check indicates that the configuration file for either dnsdist or pdns-recursor was changed but the service itself was not restarted. A CRITICAL alert is raised if the time delta between the configuration file change and service restart exceeds 24 hours.

This check is meant to be a warning alert and does not signify an error in the service.

Resolving this message

Please do not perform the steps below without contacting the Traffic team first as restarting any of these services clears the cache.

From a cumin host, restart the service mentioned in the alert on the Wikidough hosts:

sudo cumin -b 1 -s 5 'A:wikidough' 'systemctl restart dnsdist.service'

or,

sudo cumin -b 1 -s 5 'A:wikidough' 'systemctl restart pdns-recursor.service'