You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Dumps/OtherMisc

From Wikitech-static
< Dumps
Revision as of 10:31, 20 May 2019 by imported>ArielGlenn (add shorturls to this list)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

This page documents various dumpsets that are produced daily or weekly, not part of the generation of the xml/sql dumps.

All of these dumps run on database servers designated 'vslow, dumps', on a snapshot host dedicated to 'misc' dump generation (everything other than the xml/sql dumps).

The dump scripts are in our git puppet repo.

If errors are encountered when the specific cron job runs, the output is sent to ops-dumps@wikimedia.org.

  • Global block table:
    • dumped weekly
    • contains an sql-format dump of information in the global block table
    • managed by mw:Extension:GlobalBlocking) (code)
    • Issues: Unless the database server goes away during the run, or database credentials change, this job should just run
  • Cirrus search dumps:
    • dumped weekly
    • contains text indices, the file index (for commons) and the metadata index (for the entire cirrus cluster) in json format
    • run by a maintenance script in mw:Extension:CirrusSearch (code)
    • Issues: it's been quite reliable so far
  • Content Translation dumps:
    • dumped weekly
    • contains parallel corpora that can be used by developers working on machine translation.
    • run by a maintenance script in mw:Extension:ContentTranslation (code)
    • Issues: it has run out of memory when the language files being dumped have too much data; these can be split apart in order to resolve the problem. Example: see this phab task.
  • Media info:
    • dumped weekly
    • two files for each wiki, consisting of titles of media files stored locally, and those used on the project stored remotely (on Commons).
    • run by a shell wrapper around the onallwikis.py script in the operations/dumps repo (code)
    • Issues: if the database server is unavailable, up to three retries will be attempted, after which the script will give up.
  • Page titles:
    • dumped daily
    • contains a list of all page titles in the main namespace (NS 0) per project
    • run by the onallwikis.py script in the operations/dumps repo (code)
    • Issues: if the database server is unavailable, up to three retries will be attempted, after which the script will give up.
  • Media titles:
    • dumped daily
    • contains a list of all titles in the Media namespace (NS 6) per project
    • run by the onallwikis.py script in the operations/dumps repo (code)
    • Issues: if the database server is unavailable, up to three retries will be attempted, after which the script will give up.
  • Short url mappings:
    • dumped weekly
    • each line contains an entry of the form short-url|log-url
    • run by the onallwikis.py script in the operations/dumps repo (code)
    • Issues: if the database server is unavailable, up to three retries will be attempted, after which the script will give up.