You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Dumps/Snapshot hosts

From Wikitech-static
Jump to navigation Jump to search

Snapshot (XML dumps generation) cluster information

Hardware

These hosts generate the sql/XML dumps or the "misc cron" dumps. For information about the hosts that serve them, see Dumps/Dump servers.

We have one mini snapshot cluster. I would love it if another were to be set up in codfw. Someday...

In D.C.:

  • snapshot1008: primary, Dell PowerEdge R440, Debian 10 (buster), 64GB RAM, 2 Intel Xeon Gold 6130 16-core CPUs, 2 480GB SSDs in software raid 1
  • snapshot1009: testbed, Dell PowerEdge R440, Debian 10 (buster), 64GB RAM, 2 Intel Xeon Gold 6130 16-core CPUs, 2 480GB SSDs in software raid 1
  • snapshot1010: primary, Dell PowerEdge R440, Debian 10 (buster), 64GB RAM, 2 Intel Xeon Gold 6240 18-core CPUs, 2 480GB SSDs in software raid 1
  • snapshot1011: primary, Dell PowerEdge R440, Debian 10 (buster), 64GB RAM, 2 Intel Xeon Gold 6240 18-core CPUs, 2 480GB SSDs in software raid 1
  • snapshot1012: primary, Dell PowerEdge R440, Debian 10 (buster), 64GB RAM, 2 Intel Xeon Gold 6240 18-core CPUs, 2 480GB SSDs in software raid 1
  • snapshot1013: primary, Dell PowerEdge R440, Debian 10 (buster), 64GB RAM, 2 Intel Xeon Gold 6240 18-core CPUs, 2 480GB SSDs in software raid 1

Current setup

The dumps monitor which cleans up stale locks and generates the index.html file, currently runs on snapshot1010.

Worker nodes:

En wikipedia dumps normally run on snapshot1012 via the dump scheduler. Wikidata dumps run on snapshot1011 via the scheduler. All other xml/sql dumps normally run on snapshot1009 and snapshot1010, via the dump scheduler. Dumps other than the xml/sql dumps run on snapshot1008.

The testbed host is used for catchup, emergency testing of bug fixes, at scale tests of misc cron jobs, and so on.

For info on running the dumps, see Dumps#Starting_dump_runs.

Other tasks

All misc cron jobs apart from the sql/xml dump runs take place on snapshot1008. From here adds-changes dumps are run, wikidata weekly entity dumps are generated, and so on.