You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Nova Resource:Dumps/Documentation

From Wikitech-static
< Nova Resource:Dumps
Revision as of 00:11, 15 March 2019 by imported>ArielGlenn (→‎Project information)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Dumps

Description

This is a project that archives the public datasets generated by Wikimedia.

Purpose

Archive the public Wikimedia datasets.

Anticipated time span

indefinite

Project status

currently running

Contact address

https://groups.google.com/forum/#!forum/wikiteam-discuss

Willing to take contributors or not

not willing

Subject area narrow or broad

broad

Project information

Introduction

This project was created to provide a dedicated space just for transferring Wikimedia dump files to the Internet Archive. These dumps were created as a possible backup in the case of cluster-wide hardware failure, and its also often used by researchers/bots. Sometimes, these files are generated for forking of any Wikimedia project, when lots of people of a project has different aims from the original Wikimedia goal.

More information about the archiving process is available at Nova Resource:Dumps/Archive.org

Data currently being archived

Here are some information and links regarding the data that this project is archiving:

  • Wikimedia main database dumps
  • Wikimedia incremental dumps
  • Wikidata JSON dumps
  • Wikimania videos
  • OpenStreetMap datasets

Servers

  • dumps-N (where N is an integer): Main archiving servers
  • dumps-stats: Wikimedia data manipulation, including dumps above and other stuff of relevance for Wikimedia research.

Storage:

  • Before the eqiad migration we used to have a 900 GB quota (hardly sufficient for comfortable work).
  • Currently all heavy operations are conducted on /data/scratch/. We currently keep to a soft limit of using only 3 TB of space, but such disk usage is always temporary and will be deleted once the data is pushed to the Archive.
  • Everything is retained locally only for very short periods, just the time needed for packing on archive.org.

Links

Our resources:

Wikimedia data:

OSM data: