You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Maps

From Wikitech-static
Revision as of 07:51, 2 August 2016 by imported>Gehel (→‎Puppetization and Automation: Adding diagrams)
Jump to navigation Jump to search

This page describes the technical aspects of deploying Maps service on Wikimedia Foundation infrastructure.

Intro

The maps service consists of Kartotherian - a nodejs service to serve map tiles, Tilerator - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are four servers in the maps group: maps-test200{1,2,3,4}.codfw.wmnet that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are two Varnish servers in the cache_maps group: cp104{3,4}.eqiad.wmnet.

Monitoring

Importing database

maps2001 is actually not the best server for this - we should switch it around with maps2002, as it has 12 cores and 96GB RAM.

  • From http://planet.osm.org/pbf/ - find the file with the latest available date, but do NOT use "latest", as that might change at any moment.
  • curl -x webproxy.eqiad.wmnet:8080 -O http://planet.osm.org/pbf/planet-151214.osm.pbf.md5
  • curl -x webproxy.eqiad.wmnet:8080 -O http://planet.osm.org/pbf/planet-151214.osm.pbf
  • md5sum -c planet-151214.osm.pbf.md5
  • PGPASSWORD="$(< ~/osmimporter_pass)" osm2pgsql --create --slim --flat-nodes nodes.bin -C 40000 --number-processes 8 --hstore planet-151214.osm.pbf -H maps-test2001 -U osmimporter -d gis
  • additional steps to import shapes and create some indexes / functions / ... are documented in Kartotherian sources.

Notes

  • Tables are created by osm2pgsql, no need for an initial DDL script.

Kartotherian

Kartotherian servers map tiles by getting vector data from Cassandra, applying the style to it, and returning raster images. It is also capable of serving a "static image" - a map with a given width/height/scaling/zoom, and can server vector tiles directly for on-the-client rendering (WebGL maps).

To see the tiles without Varnish cache, connect to Kartotherian using an ssh tunnel, e.g. ssh -L 6533:localhost:6533 maps-test2001.codfw.wmnet and browse to http://localhost:6533

Tilerator

Tilerator is a backend vector tile pre-generation service that picks up jobs from a Redis job que, copying tiles from a Postgres DB, using sql queries into vector tiles stored in Cassandra. Postgres DBs are set up on each of the maps hosts, one master and 3 slaves. Technically, Tilerator is not even a generator, but rather a "batch copying" service, which takes tiles from one configured source (e.g. a tile generator from SQL), and puts it into another source (e.g. Cassandra tile store).

TileratorUI

TileratorUI is used to add jobs to the Tilerator job queue. Actually, TileratorUI is the same code as Tilerator, but started with a different configuration. Connect to TileratorUI using an ssh tunnel, e.g. ssh -L 6535:localhost:6535 maps-test2001.codfw.wmnet and navigating to http://localhost:6535. There, you can view any style (use set style to change it), or to schedule a job by setting all relevant fields and Control+Clicking the tile you want to schedule.

See full Tilerator documentation for all commands & parameters.

Dynamic Tile Sources

Cassandra

To create a new Cassandra data source, post something like this to the /sources as a text body. Default table name is tiles. If table or keyspace is not there, you have to use createIfMissing parameter.

v2a:
  uri: cassandra://
  params:
    keyspace: v2
    table: tiles2a
    cp: [maps-test2001.codfw.wmnet, maps-test2002.codfw.wmnet, maps-test2003.codfw.wmnet, maps-test2004.codfw.wmnet]
    username: {var: cassandra-user}
    password: {var: cassandra-pswd}
#    repfactor: 4
#    durablewrite: 0
#    createIfMissing: true

Dynamic Layer Generator

To generate just a few layers from database, create a layer filter and a layer mixer:

gentmp:
  uri: bridge://
  xml:
    npm: ["osm-bright-source", "data.xml"]
  xmlSetDataSource:
    if:
      dbname: gis
      host: ""
      type: postgis
    set:
      host: localhost
      user: {var: osmdb-user}
      password: {var: osmdb-pswd}
  xmlLayers: [admin, road]

mixtmp:
  uri: layermixer://
  params:
    sources: [{ref: v2}, {ref: gentmp}]

Once set, POST a job to copy mixtmp into the storage v2 e.g.

src=mixtmp dst=v2 baseZoom=0 fromZoom=5 beforeZoom=6 parts=10

Generating Tiles

Generate all tiles for zooms 0..7, using generator gen, saving into v3 everything including the solid tiles, up to 4 jobs per zoom.

src=gen dst=v3 parts=4 baseZoom=0 fromZoom=0 beforeZoom=8 saveSolid=1

Generated tiles only if they already exist in v2 source, and save them into v3, on zooms 8..15, 60 jobs per zoom.

src=gen dst=v3 parts=60 baseZoom=0 fromZoom=8 beforeZoom=16 sourceId=v2

Bulk Copying

The fastest way to copy a large number of tiles from one source to another is to use a large number of parts and specify saveSolid=true (skips solid tile detection). E.g. to copy all z16 tiles from v2 to v3, use

src=v2 dst=v3 zoom=16 parts=60 saveSolid=true

Postgres

  • Clear the Postgres data directory and init the database from backup (replace maps2001.codfw.wmnet by the postgres master):

rm -rf /srv/postgresql/9.4/main/* && sudo -u postgres pg_basebackup -X stream -D /srv/postgresql/9.4/main/ -h maps2001.codfw.wmnet -U replication -W

Puppetization and Automation

Some diagrams to help understand the big picture:

Prerequisites

  • passwords and postgres replication configuration is set in Ops private repo (root@palladium:~/private/hieradata/role/(codfw|eqiad)/maps/server.yaml)
  • other configuration in puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml
  • cassandra::rack is defined in puppet/hieradata/maps*.yaml
  • the maps::server, maps::master roles are associated to the maps master node (site.pp)
  • the maps::server, maps::slave roles are associated to the maps slaves node (site.pp)

Manual steps

  • To initialize the first Cassandra node, we need to add the local node to the list of seeds by manually editing /etc/cassandra/cassandra.yaml and restarting cassandra:
 seed_provider:
   # Addresses of hosts that are deemed contact points.
   # Cassandra nodes use this list of hosts to find each other and learn
   # the topology of the ring.  You must change this if you are running
   # multiple nodes!
   - class_name: org.apache.cassandra.locator.SimpleSeedProvider
     parameters:
       # seeds is actually a comma-delimited list of addresses.
       # Ex: "<ip1>,<ip2>,<ip3>"
       # Omit own host name / IP in multi-node clusters (see
       # https://phabricator.wikimedia.org/T91617).
      - seeds: 10.64.16.42,10.64.32.117,10.64.48.154 # add local node here to initialize the first Cassandra node
  • Initial data load of OSM into postgresql is done by running /usr/local/bin/osm-initial-import on the postgresql master node.
Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Maps&oldid=225359"