You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Mirrors: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Majavah
imported>JHathaway
No edit summary
Line 3: Line 3:
== mirrors.wikimedia.org ==
== mirrors.wikimedia.org ==


The Wikimedia Foundation hosts public mirrors of the [[Debian]], [[Ubuntu]] and [[Tails]] archives to give back to [[w:Free_and_open-source_software|FOSS]] projects it uses itself.
The Wikimedia Foundation hosts the following public mirrors via http:


They are currently only offered via http, although ftp and rsync may be offered in the future.
* [https://debian.org Debian]
* [https://ubuntu.com Ubuntu]
* [https://wiki.debian.org/OpenStack Debian OpenStack Backports]
* [https://tails.boum.org/ Tails]


* https://mirrors.wikimedia.org/
This service carries the dual purpose of giving back to the FOSS projects we use and as our own mirror.


== Upstream links ==
== Upstream links ==
Line 18: Line 21:


== Technical info ==
== Technical info ==
As of May 2019 they are hosted on '''sodium.wikimedia.org'''.
As of December 2021 they are hosted on '''mirror1001.wikimedia.org'''.


The server is located at the [[w:Equinix|Equinix]] data center in [[w:Ashburn, Virginia|Ashburn, Virginia, United States]] ([[eqiad]]).
The server is located at the [[w:Equinix|Equinix]] data center in [[w:Ashburn, Virginia|Ashburn, Virginia, United States]] ([[eqiad]]).
Line 30: Line 33:
There are 2 [[Icinga]] checks called "Ubuntu mirror in sync with upstream" and "Debian mirror in sync with upstream" respectively.
There are 2 [[Icinga]] checks called "Ubuntu mirror in sync with upstream" and "Debian mirror in sync with upstream" respectively.


If one of these alerts you should check on sodium what happens if you manually run the check commands, "/usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/ubuntu" / "/usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/debian".
If one of these alerts you should check on mirror1001 what happens if you manually run the check commands, "/usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/ubuntu" / "/usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/debian".


Also you can run the actual sync command manually but make sure to run it was the user "mirror", so "'''[sodium:~] $ sudo -u mirror ftpsync'''".
Also you can run the actual sync command manually but make sure to run it was the user "mirror", so "'''$ sudo -u mirror ftpsync'''".


Furthermore, you can check '''/etc/ftpsync/ftpsync.conf''' to see which upstream hosts we are rsyncing from and look at '''/var/log/ftpsync/''' to check logfiles for errors.
Furthermore, you can check '''/etc/ftpsync/ftpsync.conf''' to see which upstream hosts we are rsyncing from and look at '''/var/log/ftpsync/''' to check logfiles for errors.
For Debian you can also see if you are receiving SSH triggers: '''tail -F /var/log/auth.log | grep forced-command'''


== Contacting Wikimedia maintainers ==
== Contacting Wikimedia maintainers ==
Line 42: Line 47:
== Contacting upstream maintainers ==
== Contacting upstream maintainers ==


If the error appears to be on the upstream side you should try contacting '''mirrors@debian.org''' or '''mirrors@ubuntu.com'''.
# [https://wiki.debian.org/Teams/Mirrors Debian]
 
#* Email: [mailto:mirrors@debian.org mirrors@debian.org]
There is also the IRC channel #ubuntu-mirrors on [[Libera Chat|Freenode or Libera Chat]] (see the [https://wiki.ubuntu.com/IRC/ChannelList Ubuntu IRC channel list]).
#* IRC: irc://irc.oftc.net/debian-mirrors
#* Method: SSH Trigger, syncs are triggered by an SSH forced key command for the <code>mirror</code> user. Debian has <code>StrictHostKeyChecking no</code> set on their side, so changing our SSH host key does not affect Debian's trigger.
# [https://wiki.ubuntu.com/Mirrors Ubuntu]
#* Email: [mailto:mirrors@ubuntu.com mirrors@ubuntu.com]
#* IRC: ircs://libera.chat/ubuntu-mirrors
#* Method: Systemd Timer
# [https://wiki.debian.org/OpenStack Debian OpenStack Backports]
#* IRC: irc://irc.oftc.net#debian-openstack
#* Method: Systemd Timer
# [https://tails.boum.org/contribute/how/mirror/ Tails]
#* Email: [mailto:tails-mirrors@boum.org tails-mirrors@boum.org]
#* Method: Systemd Timer


[[Category:Runbooks]]
[[Category:Runbooks]]

Revision as of 22:29, 23 December 2021

See also: Dumps/Mirror status

mirrors.wikimedia.org

The Wikimedia Foundation hosts the following public mirrors via http:

This service carries the dual purpose of giving back to the FOSS projects we use and as our own mirror.

Upstream links

These are the Debian and Ubuntu mirror pages listing mirrors.wikimedia.org to confirm it as official:

Technical info

As of December 2021 they are hosted on mirror1001.wikimedia.org.

The server is located at the Equinix data center in Ashburn, Virginia, United States (eqiad).

The relevant Puppet class is modules/role/manifests/mirrors.pp.

The repo for the puppet code can be git cloned with "git clone https://gerrit.wikimedia.org/r/operations/puppet" from Wikimedia's Gerrit installation.

Monitoring and debugging

There are 2 Icinga checks called "Ubuntu mirror in sync with upstream" and "Debian mirror in sync with upstream" respectively.

If one of these alerts you should check on mirror1001 what happens if you manually run the check commands, "/usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/ubuntu" / "/usr/local/lib/nagios/plugins/check_apt_mirror /srv/mirrors/debian".

Also you can run the actual sync command manually but make sure to run it was the user "mirror", so "$ sudo -u mirror ftpsync".

Furthermore, you can check /etc/ftpsync/ftpsync.conf to see which upstream hosts we are rsyncing from and look at /var/log/ftpsync/ to check logfiles for errors.

For Debian you can also see if you are receiving SSH triggers: tail -F /var/log/auth.log | grep forced-command

Contacting Wikimedia maintainers

The mirror server is maintained by the Wikimedia SRE team. To report issues with mirrors.wikimedia.org, please create a ticket in Wikimedia's Phabricator and tag it with 'SRE'. You can also join the Libera.Chat IRC channels #wikimedia-operations connect and #wikimedia-tech connect.

Contacting upstream maintainers

  1. Debian
    • Email: mirrors@debian.org
    • IRC: irc://irc.oftc.net/debian-mirrors
    • Method: SSH Trigger, syncs are triggered by an SSH forced key command for the mirror user. Debian has StrictHostKeyChecking no set on their side, so changing our SSH host key does not affect Debian's trigger.
  2. Ubuntu
  3. Debian OpenStack Backports
  4. Tails