You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Mirrors: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Dzahn
(→‎Technical info: The server is located at the Equinix data center in Ashburn, Virginia, United States (eqiad).)
imported>JHathaway
No edit summary
(4 intermediate revisions by 3 users not shown)
Line 3: Line 3:
== mirrors.wikimedia.org ==
== mirrors.wikimedia.org ==


The Wikimedia Foundation hosts public mirrors of the [[Debian]], [[Ubuntu]] and [[Tails]] archives to give back to [[w:Free_and_open-source_software|FOSS]] projects it uses itself.
The Wikimedia Foundation hosts the following public mirrors via http:


They are currently only offered via http, although ftp and rsync may be offered in the future.
* [https://debian.org Debian] (Official)
* [https://ubuntu.com Ubuntu] (Official)
* [https://tails.boum.org Tails] (Official)
* [https://wiki.debian.org/OpenStack Debian OpenStack Backports] (Unofficial)


* https://mirrors.wikimedia.org/
This service carries the dual purpose of giving back to the FOSS projects we use and as our own internal mirror.


== Upstream links ==
== Technical Info ==


These are the Debian and Ubuntu mirror pages listing mirrors.wikimedia.org to confirm it as official:
# Debian
#* Publication Method:
#** DNS A/AAAA Record for ftp.us.debian.org, [https://salsa.debian.org/dsa-team/mirror/domains/-/blob/cf8fed3792929221bb5d3c3bc6a01d006368a7ab/debian.org#L1168-1169 Debian Code]
#** Official US mirror: https://www.debian.org/mirror/sponsors
#* Sync Method: SSH Trigger, syncs are triggered by an SSH forced key command for the <code>mirror</code> user. Debian has <code>StrictHostKeyChecking no</code> set on their side, so changing our SSH host key does not affect Debian's trigger.
# Ubuntu
#* Publication Method:
#** Official US mirror: https://launchpad.net/ubuntu/+mirror/wikimedia-archive
#* Sync Method: Systemd Timer
# Tails
#* Publication Method:
#** DNS A Record for dl.amnesia.boum.org.
#* Sync Method: Systemd Timer
# Debian OpenStack Backports
#* Publication Method:
#** None, not published officially, but open to the public
#* Sync Method: Systemd Timer


* https://www.debian.org/mirror/sponsors
== Implementation Info ==


* https://launchpad.net/ubuntu/+mirror/wikimedia-archive
As of December 2021 they are hosted on '''mirror1001.wikimedia.org'''.
 
== Technical info ==
As of May 2019 they are hosted on '''sodium.wikimedia.org'''.


The server is located at the [[w:Equinix|Equinix]] data center in [[w:Ashburn, Virginia|Ashburn, Virginia, United States]] ([[eqiad]]).
The server is located at the [[w:Equinix|Equinix]] data center in [[w:Ashburn, Virginia|Ashburn, Virginia, United States]] ([[eqiad]]).
Line 30: Line 46:
There are 2 [[Icinga]] checks called "Ubuntu mirror in sync with upstream" and "Debian mirror in sync with upstream" respectively.
There are 2 [[Icinga]] checks called "Ubuntu mirror in sync with upstream" and "Debian mirror in sync with upstream" respectively.


If one of these alerts you should check on sodium what happens if you manually run the check commands, "/usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/ubuntu" / "/usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/debian".
If one of these alerts you should check on mirror1001 what happens if you manually run the check commands, "/usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/ubuntu" / "/usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/debian".


Also you can run the actual sync command manually but make sure to run it was the user "mirror", so "'''[sodium:~] $ sudo -u mirror ftpsync'''".
Also you can run the actual sync command manually but make sure to run it was the user "mirror", so "'''$ sudo -u mirror ftpsync'''".


Furthermore, you can check '''/etc/ftpsync/ftpsync.conf''' to see which upstream hosts we are rsyncing from and look at '''/var/log/ftpsync/''' to check logfiles for errors.
Furthermore, you can check '''/etc/ftpsync/ftpsync.conf''' to see which upstream hosts we are rsyncing from and look at '''/var/log/ftpsync/''' to check logfiles for errors.


== Contacting wikimedia maintainers ==
For Debian you can also see if you are receiving SSH triggers: '''tail -F /var/log/auth.log | grep forced-command'''


The mirror server is maintained by the Wikimedia [[SRE]] team. To report issues with mirrors.wikimedia.org please create a ticket in Wikimedia's [[Phabricator]] and tag it with 'operations'. You can also join the [[Freenode]] IRC channels #wikimedia-operations and #wikimedia-tech.
== Contacting Wikimedia Maintainers ==


== Contacting upstream maintainers ==
The mirror server is maintained by the Wikimedia [[SRE]] team. To report issues with mirrors.wikimedia.org, please create a ticket in Wikimedia's [[Phabricator]] and tag it with 'SRE'. You can also join the Libera.Chat IRC channels {{irc|wikimedia-operations}} and {{irc|wikimedia-tech}}.


If the error appears to be on the upstream side you should try contacting '''mirrors@debian.org''' or '''mirrors@ubuntu.com'''.
== Contacting Upstream Maintainers ==


There is also channel #ubuntu-mirrors on [[Freenode]].
# [https://wiki.debian.org/Teams/Mirrors Debian]
#* Email: [mailto:mirrors@debian.org mirrors@debian.org], use this list to ask for changes to DNS records, may take a few days to obtain a reply
#* IRC: irc://irc.oftc.net/debian-mirrors
# [https://wiki.ubuntu.com/Mirrors Ubuntu]
#* Email: [mailto:mirrors@ubuntu.com mirrors@ubuntu.com]
#* IRC: ircs://libera.chat/ubuntu-mirrors
# [https://tails.boum.org/contribute/how/mirror/ Tails]
#* Email: [mailto:tails-mirrors@boum.org tails-mirrors@boum.org], use this list to ask for changes to DNS records, may take a few days to obtain a reply
# [https://wiki.debian.org/OpenStack Debian OpenStack Backports]
#* IRC: irc://irc.oftc.net#debian-openstack


[[Category:Runbooks]]
[[Category:Runbooks]]

Revision as of 22:10, 6 January 2022

See also: Dumps/Mirror status

mirrors.wikimedia.org

The Wikimedia Foundation hosts the following public mirrors via http:

This service carries the dual purpose of giving back to the FOSS projects we use and as our own internal mirror.

Technical Info

  1. Debian
    • Publication Method:
    • Sync Method: SSH Trigger, syncs are triggered by an SSH forced key command for the mirror user. Debian has StrictHostKeyChecking no set on their side, so changing our SSH host key does not affect Debian's trigger.
  2. Ubuntu
  3. Tails
    • Publication Method:
      • DNS A Record for dl.amnesia.boum.org.
    • Sync Method: Systemd Timer
  4. Debian OpenStack Backports
    • Publication Method:
      • None, not published officially, but open to the public
    • Sync Method: Systemd Timer

Implementation Info

As of December 2021 they are hosted on mirror1001.wikimedia.org.

The server is located at the Equinix data center in Ashburn, Virginia, United States (eqiad).

The relevant Puppet class is modules/role/manifests/mirrors.pp.

The repo for the puppet code can be git cloned with "git clone https://gerrit.wikimedia.org/r/operations/puppet" from Wikimedia's Gerrit installation.

Monitoring and debugging

There are 2 Icinga checks called "Ubuntu mirror in sync with upstream" and "Debian mirror in sync with upstream" respectively.

If one of these alerts you should check on mirror1001 what happens if you manually run the check commands, "/usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/ubuntu" / "/usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/debian".

Also you can run the actual sync command manually but make sure to run it was the user "mirror", so "$ sudo -u mirror ftpsync".

Furthermore, you can check /etc/ftpsync/ftpsync.conf to see which upstream hosts we are rsyncing from and look at /var/log/ftpsync/ to check logfiles for errors.

For Debian you can also see if you are receiving SSH triggers: tail -F /var/log/auth.log | grep forced-command

Contacting Wikimedia Maintainers

The mirror server is maintained by the Wikimedia SRE team. To report issues with mirrors.wikimedia.org, please create a ticket in Wikimedia's Phabricator and tag it with 'SRE'. You can also join the Libera.Chat IRC channels #wikimedia-operations connect and #wikimedia-tech connect.

Contacting Upstream Maintainers

  1. Debian
  2. Ubuntu
  3. Tails
    • Email: tails-mirrors@boum.org, use this list to ask for changes to DNS records, may take a few days to obtain a reply
  4. Debian OpenStack Backports
Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Mirrors&oldid=585766"