You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Alertmanager

From Wikitech-static
Revision as of 13:14, 26 January 2021 by imported>Filippo Giunchedi (Sketch notifications section)
Jump to navigation Jump to search

What is it?

Alertmanager is the service (and software) in charge of collecting, de-duplicating and sending notifications for alerts across WMF infrastructure. It is part of the Prometheus ecosystem and therefore Prometheus itself has native support to act as Alertmanager client. The alerts dashboard, implemented by Karma, can be reached at https://alerts.wikimedia.org/. As of Jan 2021 the dashboard is available for SSO users only, however a read-only version is possible as well.

Alertmanager is being progressively rolled out as the central place where all alerts are sent, the implementation is done in phases according to the alerting infrastructure roadmap. As of Jan 2021 LibreNMS has been fully migrated, with more services to come.

File:Alertmanager deployment jan2021.png
Alertmanager production deployment in Jan 2021

Software stack

When talking about the Alertmanager stack as a whole it is useful to list its components as deployed at Wikimedia Foundation, namely the following software is:

Notifications

As of Jan 2021, Alertmanager supports the following notification methods:

  • email - sent by Alertmanager itself
  • IRC - via the jinxer-wm bot on Freenode
  • phabricator - through @phaultfinder user
  • pages - sent via Splunk Oncall (formerly known as VictorOps)

Notification preferences are set per-team and are based on the alert' severity (respectively the team and severity labels attached to the alert)