You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Maps"

From Wikitech-static
Jump to navigation Jump to search
imported>Yurik
(intro)
imported>Yurik
Line 9: Line 9:
* [https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&c=maps+Cluster+codfw&tab=m&vn=&hide-hf=false&sh=1 Ganglia - Maps cluster] (codfw)
* [https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&c=maps+Cluster+codfw&tab=m&vn=&hide-hf=false&sh=1 Ganglia - Maps cluster] (codfw)


== Kartotherian ==
== [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] ==
Kartotherian servers map tiles by getting vector data from Cassandra, applying the [https://github.com/kartotherian/osm-bright.tm2/blob/master/README.md style] to it, and returning raster images. It is also capable of serving a "static image" - a map with a given width/height/scaling/zoom, and can server vector tiles directly for on-the-client rendering (WebGL maps).
* [Https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/kartotherian/templates/config.yaml.erb Service configuration template] (extends [[phab:diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb|base template]])
* [Https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/kartotherian/templates/config.yaml.erb Service configuration template] (extends [[phab:diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb|base template]])
* [http://git.wikimedia.org/blob/maps%2Fkartotherian.git/HEAD/sources.prod.yaml Sources configuration]
* [http://git.wikimedia.org/blob/maps%2Fkartotherian.git/HEAD/sources.prod.yaml Sources configuration]
To see the tiles without Varnish cache, connect to Kartotherian using an ssh tunnel, e.g. <code>ssh -L 6533:localhost:6533 maps-test2001.codfw.wmnet</code> and browse to http://localhost:6533  
To see the tiles without Varnish cache, connect to Kartotherian using an ssh tunnel, e.g. <code>ssh -L 6533:localhost:6533 maps-test2001.codfw.wmnet</code> and browse to http://localhost:6533  


== Tilerator ==
== [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] ==
Tilerator is a backend vector tile pre-generation service that uses a Redis job que, copying tiles from a Postgres DB, using [https://github.com/kartotherian/osm-bright.tm2source/blob/master/README.md sql queries] into vector tiles stored in Cassandra. It is controlled via POST requests from localhost. Postgres DBs are set up on each of the maps hosts, one master and 3 slaves.
* [Https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/tilerator/templates/config.yaml.erb Service configuration template] (extends [[phab:diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb|base template]])
* [Https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/tilerator/templates/config.yaml.erb Service configuration template] (extends [[phab:diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb|base template]])
* [http://git.wikimedia.org/blob/maps%2Ftilerator.git/HEAD/sources.prod.yaml Sources configuration]
* [http://git.wikimedia.org/blob/maps%2Ftilerator.git/HEAD/sources.prod.yaml Sources configuration]
Line 81: Line 83:
</syntaxhighlight>Once set, POST a job to copy <code>mixtmp</code> into the storage <code>v2</code> e.g. http://localhost:6534/add?generatorId=mixtmp&storageId=v2&baseZoom=0&fromZoom=5&beforeZoom=6&parts=10
</syntaxhighlight>Once set, POST a job to copy <code>mixtmp</code> into the storage <code>v2</code> e.g. http://localhost:6534/add?generatorId=mixtmp&storageId=v2&baseZoom=0&fromZoom=5&beforeZoom=6&parts=10


== Generating Tiles ==
=== Generating Tiles ===
Generate all tiles for zooms <code>0..7</code>, using generator <code>gen</code>, saving into <code>v3</code> everything including the solid tiles, up to 4 jobs per zoom.
Generate all tiles for zooms <code>0..7</code>, using generator <code>gen</code>, saving into <code>v3</code> everything including the solid tiles, up to 4 jobs per zoom.


Line 90: Line 92:
http://localhost:6534/add?generatorId=gen&storageId=v3&parts=60&baseZoom=0&fromZoom=8&beforeZoom=16&sourceId=v2
http://localhost:6534/add?generatorId=gen&storageId=v3&parts=60&baseZoom=0&fromZoom=8&beforeZoom=16&sourceId=v2


== Bulk Copying ==
=== Bulk Copying ===
The fastest way to copy a large number of tiles from one source to another is to use a large number of parts and specify <code>saveSolid=true</code> (skips solid tile detection). E.g. to copy all z16 tiles from v2 to v3, use<blockquote>http://localhost:6534/add?generatorId=v2&storageId=v3&zoom=16&parts=60&saveSolid=true </blockquote>
The fastest way to copy a large number of tiles from one source to another is to use a large number of parts and specify <code>saveSolid=true</code> (skips solid tile detection). E.g. to copy all z16 tiles from v2 to v3, use<blockquote>http://localhost:6534/add?generatorId=v2&storageId=v3&zoom=16&parts=60&saveSolid=true </blockquote>

Revision as of 23:49, 25 September 2015

This page describes the technical aspects of deploying Maps service on Wikimedia Foundation infrastructure.

Intro

The maps service consists of Kartotherian - a nodejs service to serve map tiles, and Tilerator - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage. There are four servers in the maps group: maps-test200{1,2,3,4}.codfw.wmnet that run both Kartotherian (port 6533, NCPU instances) and Tilerator (port 6534, half of NCPU instance). Also, there are two Varnish servers in the cache_maps group: cp104{3,4}.eqiad.wmnet.

Monitoring

Kartotherian

Kartotherian servers map tiles by getting vector data from Cassandra, applying the style to it, and returning raster images. It is also capable of serving a "static image" - a map with a given width/height/scaling/zoom, and can server vector tiles directly for on-the-client rendering (WebGL maps).

To see the tiles without Varnish cache, connect to Kartotherian using an ssh tunnel, e.g. ssh -L 6533:localhost:6533 maps-test2001.codfw.wmnet and browse to http://localhost:6533

Tilerator

Tilerator is a backend vector tile pre-generation service that uses a Redis job que, copying tiles from a Postgres DB, using sql queries into vector tiles stored in Cassandra. It is controlled via POST requests from localhost. Postgres DBs are set up on each of the maps hosts, one master and 3 slaves.

Connect to Tilerator using an ssh tunnel, e.g. ssh -L 6534:localhost:6534 maps-test2001.codfw.wmnet

See full Tilerator documentation for all commands & parameters.

Command URL
Jobs Status http://localhost:6534/
Jobs Internal Status http://localhost:6534/raw
Get a list of variables http://localhost:6534/variables
Get source configuration http://localhost:6534/sources
Add new sources POST text body in YAML to http://localhost:6534/sources

For POST requests and responses in JSON, browser extensions such as Postman tend to work well.

Dynamic Tile Sources

Cassandra

To create a new Cassandra data source, post something like this to the /sources as a text body. Default table name is tiles. If table or keyspace is not there, you have to use createIfMissing parameter.

v2a:
  uri: cassandra://
  params:
    keyspace: v2
    table: tiles2a
    cp: [maps-test2001.codfw.wmnet, maps-test2002.codfw.wmnet, maps-test2003.codfw.wmnet, maps-test2004.codfw.wmnet]
    username: {var: cassandra-user}
    password: {var: cassandra-pswd}
#    repfactor: 4
#    durablewrite: 0
#    createIfMissing: true

Dynamic Layer Generator

To generate just a few layers from database, create a layer filter and a layer mixer:

gentmp:
  uri: bridge://
  xml:
    npm: ["osm-bright-source", "data.xml"]
  xmlSetDataSource:
    if:
      dbname: gis
      host: ""
      type: postgis
    set:
      host: localhost
      user: {var: osmdb-user}
      password: {var: osmdb-pswd}
  xmlLayers: [admin, road]

mixtmp:
  uri: layermixer://
  params:
    sources: [{ref: v2}, {ref: gentmp}]

Once set, POST a job to copy mixtmp into the storage v2 e.g. http://localhost:6534/add?generatorId=mixtmp&storageId=v2&baseZoom=0&fromZoom=5&beforeZoom=6&parts=10

Generating Tiles

Generate all tiles for zooms 0..7, using generator gen, saving into v3 everything including the solid tiles, up to 4 jobs per zoom.

http://localhost:6534/add?generatorId=gen&storageId=v3&parts=4&baseZoom=0&fromZoom=0&beforeZoom=8&saveSolid=1

Generated tiles only if they already exist in v2 source, and save them into v3, on zooms 8..15, 60 jobs per zoom.

http://localhost:6534/add?generatorId=gen&storageId=v3&parts=60&baseZoom=0&fromZoom=8&beforeZoom=16&sourceId=v2

Bulk Copying

The fastest way to copy a large number of tiles from one source to another is to use a large number of parts and specify saveSolid=true (skips solid tile detection). E.g. to copy all z16 tiles from v2 to v3, use

http://localhost:6534/add?generatorId=v2&storageId=v3&zoom=16&parts=60&saveSolid=true