You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Debmonitor: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>JMeybohm
(Add a note on error when using the wrong host/FQDN)
imported>Muehlenhoff
(→‎Manually remove an image from DebMonitor: Remove direct hostname reference)
 
(5 intermediate revisions by 3 users not shown)
Line 1: Line 1:
[[File:Wikimedia + Debian.svg|thumb]]
DebMonitor is a Debian package tracker website and tool developed at the Wikimedia Foundation and used to track installed and upgradable packages across the fleet. It has multiple components.
DebMonitor is a Debian package tracker website and tool developed at the Wikimedia Foundation and used to track installed and upgradable packages across the fleet. It has multiple components.


Line 19: Line 20:
Copy-Paste from your browser will lead to HTTP 403 "Client certificate validation failed".}}
Copy-Paste from your browser will lead to HTTP 403 "Client certificate validation failed".}}
==== Manually remove a host from DebMonitor ====
==== Manually remove a host from DebMonitor ====
From one of the <code>cluster::management</code> hosts (<code>cumin[12]001</code> as of Oct. 2019) run:<syntaxhighlight lang="bash">
From one of the <code>cluster::management</code> hosts ({{CuminHosts}}) run the <code>sre.debmonitor.remove-hosts</code> cookbook. See also [[Spicerack/Cookbooks#Run_a_single_Cookbook]].
sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key
 
Alternatively it can be done manually running:<syntaxhighlight lang="bash">
sudo curl -X DELETE "https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN}" --cert "/etc/debmonitor/ssl/debmonitor_$(hostname -f | tr '.' '_').pem" --key "/etc/debmonitor/ssl/debmonitor_$(hostname -f | tr '.' '_')-key.pem"
</syntaxhighlight>
</syntaxhighlight>


Line 26: Line 29:
This will remove all tags of this image from DebMonitor.
This will remove all tags of this image from DebMonitor.


From one of the <code>builder</code> hosts (<code>deneb</code> as of April 2020) run:<syntaxhighlight lang="bash">
From one of the <code>builder</code> hosts run:<syntaxhighlight lang="bash">
# IMAGE_NAME=docker-registry.wikimedia.org/foo-bar-image
# IMAGE_NAME=docker-registry.wikimedia.org/foo-bar-image
sudo curl -X DELETE https://debmonitor.discovery.wmnet/images/${IMAGE_NAME} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key
sudo curl -X DELETE "https://debmonitor.discovery.wmnet/images/${IMAGE_NAME}" --cert "/etc/debmonitor/ssl/debmonitor__$(hostname -f | tr '.' '_').pem" --key "/etc/debmonitor/ssl/debmonitor__$(hostname -f | tr '.' '_')-key.pem"
</syntaxhighlight>
</syntaxhighlight>


[[Category:Deployment]]
[[Category:Deployment]]
[[Category:SRE Infrastructure Foundations]]

Latest revision as of 10:38, 8 July 2022

Wikimedia + Debian.svg

DebMonitor is a Debian package tracker website and tool developed at the Wikimedia Foundation and used to track installed and upgradable packages across the fleet. It has multiple components.

DebMonitor website

The DebMonitor website is a Django-based application installed in an active/passive setup with uwsgi and nginx on two dedicated Ganeti VMs and deployed via Scap. The application has two different virtual hosts, one on port 80 for Varnish and one on port 443 for the DebMonitor client (see below). The data is stored on a MySQL database hosted by the m2 cluster.

There is a weekly crontab on each host to run a garbage collection script twice a week to remove orphan objects from the database (e.g. package versions not installed anymore in any host).

DebMonitor client

The debmonitor-client Debian package is installed on all the clients and reports the installed packages to the DebMonitor active server. It reports them in three different ways:

  • A dpkg hook triggered on Dpkg::Pre-Install-Pkgs to report any change to packages. It doesn't block package actions on failure.
  • An APT hook triggered on APT::Update::Post-Invoke to report any upgradable packages. It doesn't block apt-get update on failure.
  • A daily crontab to report all installed and upgradable packages to reconcile the data in case any of the above failed.

The client authenticate with the DebMonitor server via mutual authentication using the Puppet certificate, and the server authorize the connecting host to modify only its own data.

Common commands

Manually remove a host from DebMonitor

From one of the cluster::management hosts (cumin1001.eqiad.wmnet, cumin2002.codfw.wmnet) run the sre.debmonitor.remove-hosts cookbook. See also Spicerack/Cookbooks#Run_a_single_Cookbook.

Alternatively it can be done manually running:

sudo curl -X DELETE "https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN}" --cert "/etc/debmonitor/ssl/debmonitor_$(hostname -f | tr '.' '_').pem" --key "/etc/debmonitor/ssl/debmonitor_$(hostname -f | tr '.' '_')-key.pem"

Manually remove an image from DebMonitor

This will remove all tags of this image from DebMonitor.

From one of the builder hosts run:

# IMAGE_NAME=docker-registry.wikimedia.org/foo-bar-image
sudo curl -X DELETE "https://debmonitor.discovery.wmnet/images/${IMAGE_NAME}" --cert "/etc/debmonitor/ssl/debmonitor__$(hostname -f | tr '.' '_').pem" --key "/etc/debmonitor/ssl/debmonitor__$(hostname -f | tr '.' '_')-key.pem"