You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
What is it?
Alertmanager is the service (and software) in charge of collecting, de-duplicating and sending notifications for alerts across WMF infrastructure. It is part of the Prometheus ecosystem and therefore Prometheus itself has native support to act as Alertmanager client. The alerts dashboard, implemented by Karma, can be reached at https://alerts.wikimedia.org/. As of Jan 2021 the dashboard is available for SSO users only, however a read-only version is possible as well.
Alertmanager is being progressively rolled out as the central place where all alerts are sent, the implementation is done in phases according to the alerting infrastructure roadmap. As of Jan 2021 LibreNMS has been fully migrated, with more services to come.
When talking about the Alertmanager stack as a whole it is useful to list its components as deployed at Wikimedia Foundation, namely the following software is:
- Alertmanager the daemon actually in charge of handling alerts and sending out notifications
- Karma the dashboard/UI for Alertmanager alerts, it powers https://alerts.wikimedia.org
- kthxbye implements the "acknowledgement" functionality for alerts
- alertmanager-irc-relay forwards alerts to IRC channels
- prometheus-icinga-exporter compatibility shim to forward active Icinga alerts to Alertmanager, also provides Prometheus-style metrics for Icinga
As of Jan 2021, Alertmanager supports the following notification methods:
- email - sent by Alertmanager itself
- IRC - via the
jinxer-wmbot on Freenode
- phabricator - through @phaultfinder user
- pages - sent via Splunk Oncall (formerly known as VictorOps)
Notification preferences are set per-team and are based on the alert' severity (respectively the
severity labels attached to the alert)