You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Maps: Difference between revisions
Jump to navigation
Jump to search
imported>Jgiannelos No edit summary |
imported>NPRB m (Fixed Typo. (changed me to be)) |
||
| (6 intermediate revisions by 4 users not shown) | |||
| Line 1: | Line 1: | ||
This page describes the technical aspects of deploying [[mw:Maps|Maps service]] on Wikimedia Foundation infrastructure. | This page describes the technical aspects of deploying [[mw:Maps|Maps service]] on Wikimedia Foundation infrastructure. | ||
'''The service is being actively redesigned, documentation can be found under [[Maps/v2]]''' | |||
== Intro == | == Intro == | ||
| Line 6: | Line 8: | ||
[[File:Maps_@_PI@2x.png|thumb|Kartotherian internals focus]] | [[File:Maps_@_PI@2x.png|thumb|Kartotherian internals focus]] | ||
The maps service consists of [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] - a nodejs service to serve map tiles, [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are | The maps service consists of [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] - a nodejs service to serve map tiles, [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are 20 servers in the <code>maps</code> group: <code>maps20[01-10].codfw.wmnet</code> and <code>maps10[01-10].eqiad.wmnet</code> that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are four Varnish servers per datacenter in the <code>cache_maps</code> group. | ||
== The infrastructure == | == The infrastructure == | ||
| Line 12: | Line 14: | ||
* [[Maps/Tile_storage|Tile storage]] | * [[Maps/Tile_storage|Tile storage]] | ||
* [[Maps/Kartotherian|Kartotherian]] | * [[Maps/Kartotherian|Kartotherian]] | ||
* [[Maps/ | * [[Maps/Tilerator|Tilerator]] | ||
* [[Maps/Maintenance|Maintenance]] | |||
== Miscellaneous == | == Miscellaneous == | ||
* [https://wikitech.wikimedia.org/wiki/Maps/Dynamic_tile_sources Dynamic tile sources] | * [https://wikitech.wikimedia.org/wiki/Maps/Dynamic_tile_sources Dynamic tile sources] | ||
* [[Maps/Debugging]] | |||
== Development processes == | == Development processes == | ||
* [ | * [[Maps/Services_deployment|Kartotherian/Tilerator deployment]] | ||
* [[Maps/Kartotherian packages|Kartotherian/Packages]] | |||
== Puppetization and Automation == | == Puppetization and Automation == | ||
=== Prerequisites === | === Prerequisites === | ||
* passwords and postgres replication configuration is set in Ops private repo (<code>root@ | * passwords and postgres replication configuration is set in Ops private repo (<code>root@puppetmaster1001:/srv/private/hieradata/role/(codfw|eqiad)/maps/server.yaml</code>) | ||
* other configuration in <code>puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml</code> | * other configuration in <code>puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml</code> | ||
* <code>cassandra::rack</code> is defined in <code>puppet/hieradata/hosts/maps*.yaml</code> | * <code>cassandra::rack</code> is defined in <code>puppet/hieradata/hosts/maps*.yaml</code> | ||
Latest revision as of 18:09, 16 March 2023
This page describes the technical aspects of deploying Maps service on Wikimedia Foundation infrastructure.
The service is being actively redesigned, documentation can be found under Maps/v2
Intro
The maps service consists of Kartotherian - a nodejs service to serve map tiles, Tilerator - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are 20 servers in the maps group: maps20[01-10].codfw.wmnet and maps10[01-10].eqiad.wmnet that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are four Varnish servers per datacenter in the cache_maps group.
The infrastructure
Miscellaneous
Development processes
Puppetization and Automation
Prerequisites
- passwords and postgres replication configuration is set in Ops private repo (
root@puppetmaster1001:/srv/private/hieradata/role/(codfw|eqiad)/maps/server.yaml) - other configuration in
puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml cassandra::rackis defined inpuppet/hieradata/hosts/maps*.yaml- the
role::maps::master/role::maps::slaveroles are associated to the maps nodes (site.pp)
Monitoring
- KPI dashboard
- Usage dashboard
- Usage - varnish
- Kartotherian - Grafana
- Kartotherian - Logstash
- Maps Cassandra - Logstash
- Tilerator - Grafana
- Tilerator - Logstash
Subpages
- Beta Cluster setup
- ClearTables
- ClearTables/Loading
- Debugging
- Dynamic tile sources
- External usage
- Infrastructure plans
- Kartotherian
- Kartotherian packages
- Keyspace Setup
- Maintenance
- Maps experiments
- OSM
- OSM Database Legacy
- Runbook
- Services deployment
- Tile storage
- Tilerator
- v2
- v2/Architecture
- v2/Common tasks
- v2/Infrastructure
- v2/Logs
- v2/Metrics
- v2/Troubleshooting