Dumps/Mirror status
- See also: Mirrors
Current mirrors
We keep track of current mirrors for dumps in the operations/puppet repo (
code
) in hiera; check
common/profile/dumps/distribution.yaml
for the entry
profile::dumps::distribution::mirrors
. Adding the host information here will add the mirror to the list of hosts which may rsync from our
dumps servers
.
Mirrors that are active should be marked as such; mirrors that are not yet up or which seem to have lapsed are marked as not active. The rest of the fields should be self-explanatory.
Adding a mirror
Requirements : Before you add a mirror, discuss bandwidth and storage with the contact person. Current storage requirements are listed at mirroring . Note that dumps are run twice a month and so bandwidth includes the mirror pulling the latest copy of all dumps on all projects of the types of dumps they choose to mirror.
Phab : As soon as an organization has agreed to host some dumps or datasets, add a Phabricator task for it tagged with the Dumps-generation project.
Config : Once a decision has been reached on what to be mirrored, you can add the relevant information as described under 'Current Mirrors', above.
Mirroring
: The host itself can rsync either from another mirror, if they are set up for rsync, or from us (
dumps-rsync.wikimedia.org
). We have strict bandwidth caps but we provide rsync stanzas for pulling copies of just the number of xml/sql dumps desired; see
the puppet template
for the rsync config.
If an organization wishes to pull from another mirror using this convenient setup, they can pick up the appropriate
rsync-inc-last-n.txt
file (
example
) from the mirror at the top level, and put together an appropriate rsync command on their end.
Announcing : Once this is done and the mirror is up to date, you should add it to the list of current mirrors on Meta and email an announcement to xmldatadumps-l.
Boooooo : Note that at this time there is no automatic redirection of dumps downloaders to our mirror sites .
In progress mirrors
See Phabricator for mirrors being set up/worked on.