You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Portal:Data Services: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>JJMC89
m (Reverted edits by Mgagat (talk) to last revision by Nintendofan885)
imported>SRodlund
(Fixed some names for consistency)
Line 2: Line 2:
[[File:WMCS data services.svg|right|120px|alt=WMCS data services]]
[[File:WMCS data services.svg|right|120px|alt=WMCS data services]]


'''Data Services''' include services that allow for direct access to databases and dumps, as well as web interfaces for querying and programmatic access to data stores. Services currently offered are: Wiki Replicas, ToolsDB, Wikimedia Dumps, Shared Storage, Quarry and PAWS.
'''Data Services''' includes services that allow for direct access to databases and dumps, as well as web interfaces for querying and programmatic access to data stores.  
 
Data services currently include: Wiki Replicas, ToolsDB, Wikilabels Postgres, Wikimedia Dumps, Shared Storage, CirrusSearch Elasticsearch replicas, Quarry, PAWS, and the OSM Database.


== Wiki Replicas ==
== Wiki Replicas ==
Wiki Replicas are MySQL/MariaDB databases that near-realtime replicate from the production MediaWiki databases of Wikimedia Foundation wikis. The database tables are sanitized for public use.
Wiki Replicas are MySQL/MariaDB databases that replicate near-realtime from the production MediaWiki databases of Wikimedia Foundation wikis. The database tables are sanitized for public use.


Access to the Wiki Replicas is automatically granted to all users of Toolforge. See [[Help:Toolforge/Database]] for how to access the [[Wiki replicas|Wiki Replicas]].
Access to the Wiki Replicas is automatically granted to all users of Toolforge. See [[Help:Toolforge/Database]] to learn how to access the [[Wiki replicas|Wiki Replicas]].


== ToolsDB ==
== ToolsDB ==
ToolsDB is a service that allows a Tool shared user to create and maintain a Tool specific database. See [[Help:Toolforge/Database#User databases]] for help on ToolsDB.
ToolsDB is a service that allows a Tool shared user to create and maintain a Tool specific database.  


It's acessible on the following addresses:
See [[Help:Toolforge/Database#User databases]] for help on ToolsDB.
 
ToolsDB is acessible on the following addresses:
* tools.db.svc.eqiad1.wikimedia.cloud (preferred)
* tools.db.svc.eqiad1.wikimedia.cloud (preferred)
* tools-db.tools.eqiad1.wikimedia.cloud
* tools-db.tools.eqiad1.wikimedia.cloud
Line 21: Line 25:


== Wikilabels Postgres ==
== Wikilabels Postgres ==
The Postgres database used by [[Wikilabels]] (used by Ores) is on a replicated VM cluster: [https://openstack-browser.toolforge.org/server/clouddb-wikilabels-01.clouddb-services.eqiad1.wikimedia.cloud clouddb-wikilabels-01] is the primary with [https://openstack-browser.toolforge.org/server/clouddb-wikilabels-02.clouddb-services.eqiad1.wikimedia.cloud clouddb-wikilabels-02] as the usual replica. Changes that affect the postgresql service, including upgrades/reboots, should be coordinated with Aaron Halfaker.
The[[Wikilabels]] Postgres database, used by [[mw:ORES|ORES]], is on a replicated VM cluster: [https://openstack-browser.toolforge.org/server/clouddb-wikilabels-01.clouddb-services.eqiad1.wikimedia.cloud clouddb-wikilabels-01] is the primary with [https://openstack-browser.toolforge.org/server/clouddb-wikilabels-02.clouddb-services.eqiad1.wikimedia.cloud clouddb-wikilabels-02] as the usual replica.


== Wikimedia Dumps ==
== Wikimedia Dumps ==
[https://dumps.wikimedia.org/ Wikimedia Dumps] offers a range of data downloads including full text dumps, and other datasets. Toolforge users can directly access dumps data through their Tool account, see [[Help:Toolforge/Dumps]]. Cloud VPS users can request to have the share available, see [[Help:Shared storage#.2Fpublic.2Fdumps|Help:Shared storage#/public/dumps]]. More documentation about dumps can be found at [[m:Data dumps|Data dumps]].
[https://dumps.wikimedia.org/ Wikimedia Dumps] offers a range of data downloads including full text dumps, and other datasets.  
 
Toolforge users can directly access dumps data through their Tool account. See [[Help:Toolforge/Dumps]]. Cloud VPS users can request to have the share available.See[[Help:Shared storage#.2Fpublic.2Fdumps|Help:Shared storage#/public/dumps]].  
 
More documentation about dumps can be found at [[m:Data dumps|Data dumps]].


== Shared Storage ==
== Shared Storage ==
Shared Storage is offered via [[w:Network_File_System|NFS]] for Toolforge and Cloud VPS users. Shares currently offered are described at [[Help:Shared storage]]. The Toolforge environment is setup for access by default, and other Cloud VPS projects can access some resources on special request.  
Shared Storage is offered via [[w:Network_File_System|NFS]] for Toolforge and Cloud VPS users. Shared Storage currently includes shared directories offered to VPS and ToolForge users. Tools users and Tool accounts have most of the common directories already available, and VPS users can access them on request. You can request for access to the listed shares by filing a task on Phabricator under the Data-Services and VPS-Projects projects. 
 
Currently offered shares are described at [[Help:Shared storage]]. The Toolforge environment is set up for access by default, and other Cloud VPS projects can access some resources on special request.  


[https://dumps.wikimedia.org/ Wikimedia Dumps] are also offered via the Shared Storage services, but treated as a Data Service because of their wide use.
[https://dumps.wikimedia.org/ Wikimedia Dumps] are also offered via the Shared Storage services, but treated as a Data Service because of their wide use.


== CirrusSearch elasticsearch replicas ==
== CirrusSearch Elasticsearch replicas ==
The "Cloud Elastic" servers are a replica of the [[mw:Extension:CirrusSearch|CirrusSearch]] elasticsearch indices made available to Wikimedia Cloud Services applications (both Cloud VPS and Toolforge). These servers are not generally accessible from the internet at large, rather they are only accessible through applications running inside Cloud Services. Applications can use the full power of the [https://www.elastic.co/guide/en/elasticsearch/reference/6.5/query-dsl.html elasticsearch search API's] to query the search indices in ways that CirrusSearch does not expose directly on the wiki's themselves.
The "Cloud Elastic" servers are a replica of the [[mw:Extension:CirrusSearch|CirrusSearch]] Elasticsearch indices made available to Wikimedia Cloud Services applications (both Cloud VPS and Toolforge). These servers are not generally accessible from the internet at large, rather they are only accessible through applications running inside Wikimedia Cloud Services. Applications can use the full power of the [https://www.elastic.co/guide/en/elasticsearch/reference/6.5/query-dsl.html elasticsearch search API's] to query the search indices in ways that CirrusSearch does not expose directly on the wikis themselves.


See [[Help:CirrusSearch elasticsearch replicas]] for more details on how to access and use the Cloud Elastic service.
See [[Help:CirrusSearch elasticsearch replicas]] for more details on how to access and use the Cloud Elastic service.


== Quarry ==
== Quarry ==
[https://quarry.wmflabs.org/ Quarry] is a graphical web interface that allows users to query the Wiki Replicas with SQL. It only needs a Wikimedia account to login, and is extensively used by analysts, researchers, and people of all experience levels to easily access the databases. See [[m:Research:Quarry]] for help.
[https://quarry.wmflabs.org/ Quarry] is a graphical web interface that allows users to query the Wiki Replicas with SQL. Quarry requires a Wikimedia SUL account to login and is extensively used by analysts, researchers, and people of all experience levels to easily access the databases. See [[m:Research:Quarry]] for help.


== PAWS ==
== PAWS ==
[https://hub.paws.wmcloud.org PAWS] is a [https://jupyter.org Jupyter] notebooks on the cloud service that hosts python notebooks and a terminal accessible through a web browser. It also only requires a Wikimedia account to login, and allows for access to the Wiki Replicas, ToolsDB and Dumps. See [[PAWS]] for help.
[[PAWS]] is a [https://jupyter.org Jupyter] notebooks installation hosted by Wikimedia Cloud Services that hosts Python notebooks and a terminal accessible through a web browser. PAWS requires a Wikimedia SUL account to login. You can access Wiki Replicas, ToolsDB and Dumps with PAWS.


== OSM Database ==
== OSM Database ==


We provide a clone of the OSM (OpenStreetMap) database for usage inside Toolforge and Cloud VPS. See [[Help:Toolforge/Database#Connecting to OSM via the official CLI PostgreSQL]] and [[Openstreetmap Databases]] for more information.
Wikimedia Cloud Services provides a clone of the OSM (OpenStreetMap) database for usage inside Toolforge and Cloud VPS. See [[Help:Toolforge/Database#Connecting to OSM via the official CLI PostgreSQL]] and [[Openstreetmap Databases]] for more information.


== See also ==
== See also ==

Revision as of 21:14, 26 October 2020

WMCS data services

Data Services includes services that allow for direct access to databases and dumps, as well as web interfaces for querying and programmatic access to data stores.

Data services currently include: Wiki Replicas, ToolsDB, Wikilabels Postgres, Wikimedia Dumps, Shared Storage, CirrusSearch Elasticsearch replicas, Quarry, PAWS, and the OSM Database.

Wiki Replicas

Wiki Replicas are MySQL/MariaDB databases that replicate near-realtime from the production MediaWiki databases of Wikimedia Foundation wikis. The database tables are sanitized for public use.

Access to the Wiki Replicas is automatically granted to all users of Toolforge. See Help:Toolforge/Database to learn how to access the Wiki Replicas.

ToolsDB

ToolsDB is a service that allows a Tool shared user to create and maintain a Tool specific database.

See Help:Toolforge/Database#User databases for help on ToolsDB.

ToolsDB is acessible on the following addresses:

  • tools.db.svc.eqiad1.wikimedia.cloud (preferred)
  • tools-db.tools.eqiad1.wikimedia.cloud

It used to run on labsdb1005 and got migrated into a Cloud VPS VM called clouddb1001 in the clouddb-services project (more details about the migration are available in phab:T208754 phab:T193264).

You can verify the service status and the availability report in Icinga. Active checks are carried out by Toolschecker upon request by Icinga.

Wikilabels Postgres

TheWikilabels Postgres database, used by ORES, is on a replicated VM cluster: clouddb-wikilabels-01 is the primary with clouddb-wikilabels-02 as the usual replica.

Wikimedia Dumps

Wikimedia Dumps offers a range of data downloads including full text dumps, and other datasets.

Toolforge users can directly access dumps data through their Tool account. See Help:Toolforge/Dumps. Cloud VPS users can request to have the share available.SeeHelp:Shared storage#/public/dumps.

More documentation about dumps can be found at Data dumps.

Shared Storage

Shared Storage is offered via NFS for Toolforge and Cloud VPS users. Shared Storage currently includes shared directories offered to VPS and ToolForge users. Tools users and Tool accounts have most of the common directories already available, and VPS users can access them on request. You can request for access to the listed shares by filing a task on Phabricator under the Data-Services and VPS-Projects projects.

Currently offered shares are described at Help:Shared storage. The Toolforge environment is set up for access by default, and other Cloud VPS projects can access some resources on special request.

Wikimedia Dumps are also offered via the Shared Storage services, but treated as a Data Service because of their wide use.

CirrusSearch Elasticsearch replicas

The "Cloud Elastic" servers are a replica of the CirrusSearch Elasticsearch indices made available to Wikimedia Cloud Services applications (both Cloud VPS and Toolforge). These servers are not generally accessible from the internet at large, rather they are only accessible through applications running inside Wikimedia Cloud Services. Applications can use the full power of the elasticsearch search API's to query the search indices in ways that CirrusSearch does not expose directly on the wikis themselves.

See Help:CirrusSearch elasticsearch replicas for more details on how to access and use the Cloud Elastic service.

Quarry

Quarry is a graphical web interface that allows users to query the Wiki Replicas with SQL. Quarry requires a Wikimedia SUL account to login and is extensively used by analysts, researchers, and people of all experience levels to easily access the databases. See m:Research:Quarry for help.

PAWS

PAWS is a Jupyter notebooks installation hosted by Wikimedia Cloud Services that hosts Python notebooks and a terminal accessible through a web browser. PAWS requires a Wikimedia SUL account to login. You can access Wiki Replicas, ToolsDB and Dumps with PAWS.

OSM Database

Wikimedia Cloud Services provides a clone of the OSM (OpenStreetMap) database for usage inside Toolforge and Cloud VPS. See Help:Toolforge/Database#Connecting to OSM via the official CLI PostgreSQL and Openstreetmap Databases for more information.

See also