You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Mirrors: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>JHathaway
No edit summary
imported>JHathaway
No edit summary
Line 5: Line 5:
The Wikimedia Foundation hosts the following public mirrors via http:
The Wikimedia Foundation hosts the following public mirrors via http:


* [https://debian.org Debian]
* [https://debian.org Debian] (Official)
* [https://ubuntu.com Ubuntu]
* [https://ubuntu.com Ubuntu] (Official)
* [https://wiki.debian.org/OpenStack Debian OpenStack Backports]
* [https://tails.boum.org Tails] (Official)
* [https://tails.boum.org/ Tails]
* [https://wiki.debian.org/OpenStack Debian OpenStack Backports] (Unofficial)


This service carries the dual purpose of giving back to the FOSS projects we use and as our own mirror.
This service carries the dual purpose of giving back to the FOSS projects we use and as our own internal mirror.


== Upstream links ==
== Technical Info ==


These are the Debian and Ubuntu mirror pages listing mirrors.wikimedia.org to confirm it as official:
# Debian
#* Publication Method:
#** DNS A/AAAA Record for ftp.us.debian.org, [https://salsa.debian.org/dsa-team/mirror/domains/-/blob/cf8fed3792929221bb5d3c3bc6a01d006368a7ab/debian.org#L1168-1169 Debian Code]
#** Official US mirror: https://www.debian.org/mirror/sponsors
#* Sync Method: SSH Trigger, syncs are triggered by an SSH forced key command for the <code>mirror</code> user. Debian has <code>StrictHostKeyChecking no</code> set on their side, so changing our SSH host key does not affect Debian's trigger.
# Ubuntu
#* Publication Method:
#** Official US mirror: https://launchpad.net/ubuntu/+mirror/wikimedia-archive
#* Sync Method: Systemd Timer
# Tails
#* Publication Method:
#** DNS A Record for dl.amnesia.boum.org.
#* Sync Method: Systemd Timer
# Debian OpenStack Backports
#* Publication Method:
#** None, not published officially, but open to the public
#* Sync Method: Systemd Timer


* https://www.debian.org/mirror/sponsors
== Implementation Info ==


* https://launchpad.net/ubuntu/+mirror/wikimedia-archive
== Technical info ==
As of December 2021 they are hosted on '''mirror1001.wikimedia.org'''.
As of December 2021 they are hosted on '''mirror1001.wikimedia.org'''.


Line 41: Line 54:
For Debian you can also see if you are receiving SSH triggers: '''tail -F /var/log/auth.log | grep forced-command'''
For Debian you can also see if you are receiving SSH triggers: '''tail -F /var/log/auth.log | grep forced-command'''


== Contacting Wikimedia maintainers ==
== Contacting Wikimedia Maintainers ==


The mirror server is maintained by the Wikimedia [[SRE]] team. To report issues with mirrors.wikimedia.org, please create a ticket in Wikimedia's [[Phabricator]] and tag it with 'SRE'. You can also join the Libera.Chat IRC channels {{irc|wikimedia-operations}} and {{irc|wikimedia-tech}}.
The mirror server is maintained by the Wikimedia [[SRE]] team. To report issues with mirrors.wikimedia.org, please create a ticket in Wikimedia's [[Phabricator]] and tag it with 'SRE'. You can also join the Libera.Chat IRC channels {{irc|wikimedia-operations}} and {{irc|wikimedia-tech}}.


== Contacting upstream maintainers ==
== Contacting Upstream Maintainers ==


# [https://wiki.debian.org/Teams/Mirrors Debian]
# [https://wiki.debian.org/Teams/Mirrors Debian]
#* Email: [mailto:mirrors@debian.org mirrors@debian.org]
#* Email: [mailto:mirrors@debian.org mirrors@debian.org], use this list to ask for changes to DNS records, may take a few days to obtain a reply
#* IRC: irc://irc.oftc.net/debian-mirrors
#* IRC: irc://irc.oftc.net/debian-mirrors
#* Method: SSH Trigger, syncs are triggered by an SSH forced key command for the <code>mirror</code> user. Debian has <code>StrictHostKeyChecking no</code> set on their side, so changing our SSH host key does not affect Debian's trigger.
# [https://wiki.ubuntu.com/Mirrors Ubuntu]
# [https://wiki.ubuntu.com/Mirrors Ubuntu]
#* Email: [mailto:mirrors@ubuntu.com mirrors@ubuntu.com]
#* Email: [mailto:mirrors@ubuntu.com mirrors@ubuntu.com]
#* IRC: ircs://libera.chat/ubuntu-mirrors
#* IRC: ircs://libera.chat/ubuntu-mirrors
#* Method: Systemd Timer
# [https://tails.boum.org/contribute/how/mirror/ Tails]
#* Email: [mailto:tails-mirrors@boum.org tails-mirrors@boum.org], use this list to ask for changes to DNS records, may take a few days to obtain a reply
# [https://wiki.debian.org/OpenStack Debian OpenStack Backports]
# [https://wiki.debian.org/OpenStack Debian OpenStack Backports]
#* IRC: irc://irc.oftc.net#debian-openstack
#* IRC: irc://irc.oftc.net#debian-openstack
#* Method: Systemd Timer
# [https://tails.boum.org/contribute/how/mirror/ Tails]
#* Email: [mailto:tails-mirrors@boum.org tails-mirrors@boum.org]
#* Method: Systemd Timer


[[Category:Runbooks]]
[[Category:Runbooks]]

Revision as of 22:10, 6 January 2022

See also: Dumps/Mirror status

mirrors.wikimedia.org

The Wikimedia Foundation hosts the following public mirrors via http:

This service carries the dual purpose of giving back to the FOSS projects we use and as our own internal mirror.

Technical Info

  1. Debian
    • Publication Method:
    • Sync Method: SSH Trigger, syncs are triggered by an SSH forced key command for the mirror user. Debian has StrictHostKeyChecking no set on their side, so changing our SSH host key does not affect Debian's trigger.
  2. Ubuntu
  3. Tails
    • Publication Method:
      • DNS A Record for dl.amnesia.boum.org.
    • Sync Method: Systemd Timer
  4. Debian OpenStack Backports
    • Publication Method:
      • None, not published officially, but open to the public
    • Sync Method: Systemd Timer

Implementation Info

As of December 2021 they are hosted on mirror1001.wikimedia.org.

The server is located at the Equinix data center in Ashburn, Virginia, United States (eqiad).

The relevant Puppet class is modules/role/manifests/mirrors.pp.

The repo for the puppet code can be git cloned with "git clone https://gerrit.wikimedia.org/r/operations/puppet" from Wikimedia's Gerrit installation.

Monitoring and debugging

There are 2 Icinga checks called "Ubuntu mirror in sync with upstream" and "Debian mirror in sync with upstream" respectively.

If one of these alerts you should check on mirror1001 what happens if you manually run the check commands, "/usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/ubuntu" / "/usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/debian".

Also you can run the actual sync command manually but make sure to run it was the user "mirror", so "$ sudo -u mirror ftpsync".

Furthermore, you can check /etc/ftpsync/ftpsync.conf to see which upstream hosts we are rsyncing from and look at /var/log/ftpsync/ to check logfiles for errors.

For Debian you can also see if you are receiving SSH triggers: tail -F /var/log/auth.log | grep forced-command

Contacting Wikimedia Maintainers

The mirror server is maintained by the Wikimedia SRE team. To report issues with mirrors.wikimedia.org, please create a ticket in Wikimedia's Phabricator and tag it with 'SRE'. You can also join the Libera.Chat IRC channels #wikimedia-operations connect and #wikimedia-tech connect.

Contacting Upstream Maintainers

  1. Debian
  2. Ubuntu
  3. Tails
    • Email: tails-mirrors@boum.org, use this list to ask for changes to DNS records, may take a few days to obtain a reply
  4. Debian OpenStack Backports