You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Maps: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Yurik
(configuration & source links)
imported>NPRB
m (Fixed Typo. (changed me to be))
 
(50 intermediate revisions by 14 users not shown)
Line 1: Line 1:
This page describes the technical aspects of deploying [https://www.mediawiki.org/wiki/Maps Maps service] on Wikimedia Foundation infrastructure.
This page describes the technical aspects of deploying [[mw:Maps|Maps service]] on Wikimedia Foundation infrastructure.  


== Monitoring ==
'''The service is being actively redesigned, documentation can be found under [[Maps/v2]]'''
* [http://searchdata.wmflabs.org/maps/ Usage dashboard]
* [https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=cpu_report&s=by+name&c=Maps%2520caches%2520eqiad&tab=m&vn=&hide-hf=false Ganglia - Maps Varnish cluster] (eqiad)
* [https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&c=maps+Cluster+codfw&tab=m&vn=&hide-hf=false&sh=1 Ganglia - Maps cluster] (codfw)


== Kartotherian ==
== Intro ==
* [Https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/kartotherian/templates/config.yaml.erb Service configuration template] (extends [[phab:diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb|base template]])
[[File:Maps-components.png|thumb|Maps service component diagram]]
* [http://git.wikimedia.org/blob/maps%2Fkartotherian.git/HEAD/sources.prod.yaml Sources configuration]
[[File:Maps-deployment.png|thumb|Maps service deployment diagram]]
To see the tiles without Varnish cache, connect to Kartotherian using an ssh tunnel, e.g. <code>ssh -L 6533:localhost:6533 maps-test2001.codfw.wmnet</code> and browse to http://localhost:6533
[[File:Maps_@_PI@2x.png|thumb|Kartotherian internals focus]]


== Tilerator ==
The maps service consists of [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] - a nodejs service to serve map tiles, [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are 20 servers in the <code>maps</code> group: <code>maps20[01-10].codfw.wmnet</code> and <code>maps10[01-10].eqiad.wmnet</code> that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are four Varnish servers per datacenter in the <code>cache_maps</code> group.
* [Https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/tilerator/templates/config.yaml.erb Service configuration template] (extends [[phab:diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb|base template]])
* [http://git.wikimedia.org/blob/maps%2Ftilerator.git/HEAD/sources.prod.yaml Sources configuration]
Connect to Tilerator using an ssh tunnel, e.g. <code>ssh -L 6534:localhost:6534 maps-test2001.codfw.wmnet</code>


See full [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator documentation] for all commands & parameters.
== The infrastructure ==
{| class="wikitable"
* [[Maps/OSM Database|OSM Database]]
!Command
* [[Maps/Tile_storage|Tile storage]]
!URL
* [[Maps/Kartotherian|Kartotherian]]
|-
* [[Maps/Tilerator|Tilerator]]
|Jobs Status
* [[Maps/Maintenance|Maintenance]]
|http://localhost:6534/
|-
|Jobs Internal Status
|http://localhost:6534/raw
|-
|Get a list of variables
|http://localhost:6534/variables
|-
|Get source configuration
|http://localhost:6534/sources
|-
|Add new sources
|POST text body in YAML to [http://localhost:6534/variable http://localhost:6534/sources]
|}
For POST requests and responses in JSON, browser extensions such as [https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop?hl=en Postman] tend to work well.


=== Dynamic Tile Sources ===
== Miscellaneous ==
* [https://wikitech.wikimedia.org/wiki/Maps/Dynamic_tile_sources Dynamic tile sources]
* [[Maps/Debugging]]


==== Cassandra ====
== Development processes ==
To create a new Cassandra data source, post something like this to the /sources as a text body. Default table name is <code>tiles</code>. If table or keyspace is not there, you have to use <code>createIfMissing</code> parameter.<syntaxhighlight lang="yaml">
* [[Maps/Services_deployment|Kartotherian/Tilerator deployment]]
v2a:
* [[Maps/Kartotherian packages|Kartotherian/Packages]]
  uri: cassandra://
  params:
    keyspace: v2
    table: tiles2a
    cp: [maps-test2001.codfw.wmnet, maps-test2002.codfw.wmnet, maps-test2003.codfw.wmnet, maps-test2004.codfw.wmnet]
    username: {var: cassandra-user}
    password: {var: cassandra-pswd}
#    repfactor: 4
#    durablewrite: 0
#    createIfMissing: true
</syntaxhighlight>


==== Dynamic Layer Generator ====
== Puppetization and Automation ==
To generate just a few layers from database, create a layer filter and a layer mixer:<syntaxhighlight lang="yaml">
=== Prerequisites ===
gentmp:
* passwords and postgres replication configuration is set in Ops private repo (<code>root@puppetmaster1001:/srv/private/hieradata/role/(codfw|eqiad)/maps/server.yaml</code>)
  uri: bridge://
* other configuration in <code>puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml</code>
  xml:
* <code>cassandra::rack</code> is defined in <code>puppet/hieradata/hosts/maps*.yaml</code>
    npm: ["osm-bright-source", "data.xml"]
* the <code>role::maps::master</code> / <code>role::maps::slave</code> roles are associated to the maps nodes (site.pp)
  xmlSetDataSource:
    if:
      dbname: gis
      host: ""
      type: postgis
    set:
      host: localhost
      user: {var: osmdb-user}
      password: {var: osmdb-pswd}
  xmlLayers: [admin, road]


mixtmp:
== Monitoring ==
  uri: layermixer://
* [https://grafana.wikimedia.org/dashboard/db/interactive-team-kpi KPI dashboard]
  params:
* [http://discovery.wmflabs.org/maps/ Usage dashboard]
    sources: [{ref: v2}, {ref: gentmp}]
* [https://grafana.wikimedia.org/dashboard/db/service-maps-varnish Usage - varnish]
</syntaxhighlight>Once set, POST a job to copy <code>mixtmp</code> into the storage <code>v2</code> e.g. http://localhost:6534/add?generatorId=mixtmp&storageId=v2&baseZoom=0&fromZoom=5&beforeZoom=6&parts=10
* [https://grafana.wikimedia.org/dashboard/db/service-kartotherian Kartotherian - Grafana]
 
* [https://logstash.wikimedia.org/#/dashboard/elasticsearch/kartotherian Kartotherian - Logstash]
== Generating Tiles ==
* [https://logstash.wikimedia.org/#/dashboard/elasticsearch/maps-cassandra Maps Cassandra - Logstash]
Generate all tiles for zooms <code>0..7</code>, using generator <code>gen</code>, saving into <code>v3</code> everything including the solid tiles, up to 4 jobs per zoom.
* [https://grafana.wikimedia.org/dashboard/db/service-tilerator Tilerator - Grafana]
 
* [https://logstash.wikimedia.org/#/dashboard/elasticsearch/tilerator Tilerator - Logstash]
http://localhost:6534/add?generatorId=gen&storageId=v3&parts=4&baseZoom=0&fromZoom=0&beforeZoom=8&saveSolid=1
 
Generated tiles only if they already exist in <code>v2</code> source, and save them into <code>v3</code>, on zooms <code>8..15</code>, 60 jobs per zoom.
 
http://localhost:6534/add?generatorId=gen&storageId=v3&parts=60&baseZoom=0&fromZoom=8&beforeZoom=16&sourceId=v2


== Bulk Copying ==
== Subpages ==
The fastest way to copy a large number of tiles from one source to another is to use a large number of parts and specify <code>saveSolid=true</code> (skips solid tile detection). E.g. to copy all z16 tiles from v2 to v3, use<blockquote>http://localhost:6534/add?generatorId=v2&storageId=v3&zoom=16&parts=60&saveSolid=true </blockquote>
{{Special:PrefixIndex/{{PAGENAME}}/|hideredirects=1|stripprefix=1}}

Latest revision as of 18:09, 16 March 2023

This page describes the technical aspects of deploying Maps service on Wikimedia Foundation infrastructure.

The service is being actively redesigned, documentation can be found under Maps/v2

Intro

Maps service component diagram
Maps service deployment diagram
Error creating thumbnail: File with dimensions greater than 12.5 MP
Kartotherian internals focus

The maps service consists of Kartotherian - a nodejs service to serve map tiles, Tilerator - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are 20 servers in the maps group: maps20[01-10].codfw.wmnet and maps10[01-10].eqiad.wmnet that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are four Varnish servers per datacenter in the cache_maps group.

The infrastructure

Miscellaneous

Development processes

Puppetization and Automation

Prerequisites

  • passwords and postgres replication configuration is set in Ops private repo (root@puppetmaster1001:/srv/private/hieradata/role/(codfw|eqiad)/maps/server.yaml)
  • other configuration in puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml
  • cassandra::rack is defined in puppet/hieradata/hosts/maps*.yaml
  • the role::maps::master / role::maps::slave roles are associated to the maps nodes (site.pp)

Monitoring

Subpages

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Maps&oldid=599253"