You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
LibreNMS is an autodiscovering PHP/MySQL/SNMP based network monitoring which includes support for a wide range of network hardware and operating systems including Cisco, Linux, Juniper, Foundry, and many more.
LibreNMS is a community-based fork of the last GPL-licensed version of Observium.
Currently hosted on netmon1002 and netmon2001
Replaces Observium which ran on Streber.
- Software is not installed via Debian package
- Software installed in:
- RRD data stored in:
- User creds are stored in MySQL:
# grep auth_mechanism /srv/deployment/librenms/librenms/config.php
- Authentication is done via LDAP
Add a device to LibreNMS
Configure the read only v2c SNMP community on the device
And use the device FQDN, keep all the other fields as it (and do not force add it). Note: because of a bug, set port to "161".
The device should be discovered and polled in the next 10min.
$ ssh librenms.wikimedia.org $ cd /srv/deployment/librenms/librenms $ sudo -u librenms php addhost.php <fqdn> Added device <fqdn> (XXX) $ sudo -u librenms php discovery.php -h XXX && sudo -u librenms php poller.php -h XXX
Let's assume your remote is configured like the following. And we're tracking new versions in different branches.
origin ssh://<username>@gerrit.wikimedia.org:29418/operations/software/librenms (fetch) origin ssh://<username>@gerrit.wikimedia.org:29418/operations/software/librenms (push) upstream https://github.com/librenms/librenms.git (fetch) upstream https://github.com/librenms/librenms.git (push)
git fetch upstream
git checkout -b upstream-<version> <version>
(Verify you're in the
composer install --no-dev (your will be prompted for any missing php requirements)
git add -f vendor
git commit -m "Add composer requirements for LibreNMS<version>"
[global] git_repo: librenms/librenms git_deploy_dir: /srv/deployment ssh_user: deploy-librenms dsh_targets: /etc/dsh/group/librenms git_submodules: False lock_file: /tmp/scap.librenms.lock config_deploy: False keyholder_key: deploy_librenms
git add scap
git commit -m "Add Scap config"
git push origin upstream-<version>
git fetch origin
git status - note the current branch
git checkout upstream-<version>
scap deploy Upgrade LibreNMS to <version> - <task>
Run sudo -u librenms ./daily.sh
git fetch origin
git checkout <previous branch>
scap deploy Rollback LibreNMS to <version> - <task>
Check the logs
LibreNMS logs in 4 different locations:
It would be great to have the first 3 in a single location.
Mass update PDU alerting thresholds
PDUs have automatically generated thresholds, the query bellow sets sane defaults to eqiad/codfw PDUs.
UPDATE librenms.sensors JOIN librenms.devices ON sensors.device_id = devices.device_id AND sensor_class = 'power' AND sensor_descr like "Phase%" AND (hostname like "%eqiad%" or hostname like "%codfw%" ) SET sensors.sensor_custom = 'Yes', sensor_limit = 1400
UPDATE librenms.sensors JOIN librenms.devices ON sensors.device_id = devices.device_id AND sensor_class = 'current' AND sensor_descr like "Phase%" AND (hostname like "%eqiad%" or hostname like "%codfw%" ) SET sensors.sensor_custom = 'Yes', sensor_limit = 12
LibreNMS can group interfaces based on their description's prefix, for example "Transit:", "Peering:". Which is shown under the "ports" dropdown.
Prefixes not shown in the dropdown are still reachable by editing the URL, for example:
LibreNMS IRC bot named librenms-wmf outputs the alerts and recoveries on the -operations channel. This to help correlate series of alerts between different monitoring tools.
More details about the alerts can be found on https://librenms.wikimedia.org/alerts/ .
If the bot misbehaves or is too noisy and needs to be stopped, three options:
- Downtime the host in https://librenms.wikimedia.org/alert-schedule/
- Turn off "IRC Transport" in https://librenms.wikimedia.org/settings/sub=alerting/
- ssh to librenms.wikimedia.org and turn of the service
sudo service librenms-ircbot stop
Then file a task to track the issue.
If IRC alerting is not working:
- Make sure the bot is running
- Make sure the fifo file librenms/.ircbot.alert (created by the IRC bot) is writable by www-data (the LibreNMS app)