You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Maps: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Jgiannelos
No edit summary
imported>Effie Mouzeli
No edit summary
 
(4 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This page describes the technical aspects of deploying [[mw:Maps|Maps service]] on Wikimedia Foundation infrastructure.
This page describes the technical aspects of deploying [[mw:Maps|Maps service]] on Wikimedia Foundation infrastructure.  
 
'''The service is being actively redesigned, documentation can me found under [[Maps/v2]]'''


== Intro ==
== Intro ==
Line 6: Line 8:
[[File:Maps_@_PI@2x.png|thumb|Kartotherian internals focus]]
[[File:Maps_@_PI@2x.png|thumb|Kartotherian internals focus]]


The maps service consists of [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] - a nodejs service to serve map tiles, [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are four servers in the <code>maps</code> group: <code>maps-test200{1,2,3,4}.codfw.wmnet</code> that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are four Varnish servers per datacenter in the <code>cache_maps</code> group.
The maps service consists of [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] - a nodejs service to serve map tiles, [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are 20 servers in the <code>maps</code> group: <code>maps20[01-10].codfw.wmnet</code> and <code>maps10[01-10].eqiad.wmnet</code> that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are four Varnish servers per datacenter in the <code>cache_maps</code> group.


== The infrastructure ==
== The infrastructure ==
Line 12: Line 14:
* [[Maps/Tile_storage|Tile storage]]
* [[Maps/Tile_storage|Tile storage]]
* [[Maps/Kartotherian|Kartotherian]]
* [[Maps/Kartotherian|Kartotherian]]
* [[Maps/Kartotherian|Tilerator]]
* [[Maps/Tilerator|Tilerator]]
* [[Maps/Maintenance|Maintenance]]


== Miscellaneous ==
== Miscellaneous ==
* [https://wikitech.wikimedia.org/wiki/Maps/Dynamic_tile_sources Dynamic tile sources]
* [https://wikitech.wikimedia.org/wiki/Maps/Dynamic_tile_sources Dynamic tile sources]
* [[Maps/Debugging]]


== Development processes ==
== Development processes ==
Line 22: Line 26:
== Puppetization and Automation ==
== Puppetization and Automation ==
=== Prerequisites ===
=== Prerequisites ===
* passwords and postgres replication configuration is set in Ops private repo (<code>root@palladium:~/private/hieradata/role/(codfw|eqiad)/maps/server.yaml</code>)
* passwords and postgres replication configuration is set in Ops private repo (<code>root@puppetmaster1001:/srv/private/hieradata/role/(codfw|eqiad)/maps/server.yaml</code>)
* other configuration in <code>puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml</code>
* other configuration in <code>puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml</code>
* <code>cassandra::rack</code> is defined in <code>puppet/hieradata/hosts/maps*.yaml</code>
* <code>cassandra::rack</code> is defined in <code>puppet/hieradata/hosts/maps*.yaml</code>

Latest revision as of 10:42, 13 October 2021

This page describes the technical aspects of deploying Maps service on Wikimedia Foundation infrastructure.

The service is being actively redesigned, documentation can me found under Maps/v2

Intro

Maps service component diagram
Maps service deployment diagram
File:Maps @ PI@2x.png
Kartotherian internals focus

The maps service consists of Kartotherian - a nodejs service to serve map tiles, Tilerator - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are 20 servers in the maps group: maps20[01-10].codfw.wmnet and maps10[01-10].eqiad.wmnet that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are four Varnish servers per datacenter in the cache_maps group.

The infrastructure

Miscellaneous

Development processes

Puppetization and Automation

Prerequisites

  • passwords and postgres replication configuration is set in Ops private repo (root@puppetmaster1001:/srv/private/hieradata/role/(codfw|eqiad)/maps/server.yaml)
  • other configuration in puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml
  • cassandra::rack is defined in puppet/hieradata/hosts/maps*.yaml
  • the role::maps::master / role::maps::slave roles are associated to the maps nodes (site.pp)

Monitoring

Subpages