imported>Gehel |
imported>Effie Mouzeli |
| (17 intermediate revisions by 10 users not shown) |
| Line 1: |
Line 1: |
| This page describes the technical aspects of deploying [[mw:Maps|Maps service]] on Wikimedia Foundation infrastructure. | | This page describes the technical aspects of deploying [[mw:Maps|Maps service]] on Wikimedia Foundation infrastructure. |
| | |
| | '''The service is being actively redesigned, documentation can me found under [[Maps/v2]]''' |
|
| |
|
| == Intro == | | == Intro == |
| [[File:Maps-components.png|thumb|Maps service component diagram]] | | [[File:Maps-components.png|thumb|Maps service component diagram]] |
| [[File:Maps-deployment.png|thumb|Maps service deployment diagram]] | | [[File:Maps-deployment.png|thumb|Maps service deployment diagram]] |
| The maps service consists of [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] - a nodejs service to serve map tiles, [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are four servers in the <code>maps</code> group: <code>maps-test200{1,2,3,4}.codfw.wmnet</code> that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are two Varnish servers in the <code>cache_maps</code> group: <code>cp104{3,4}.eqiad.wmnet</code>. | | [[File:Maps_@_PI@2x.png|thumb|Kartotherian internals focus]] |
| | |
| | The maps service consists of [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] - a nodejs service to serve map tiles, [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are 20 servers in the <code>maps</code> group: <code>maps20[01-10].codfw.wmnet</code> and <code>maps10[01-10].eqiad.wmnet</code> that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are four Varnish servers per datacenter in the <code>cache_maps</code> group. |
| | |
| | == The infrastructure == |
| | * [[Maps/OSM Database|OSM Database]] |
| | * [[Maps/Tile_storage|Tile storage]] |
| | * [[Maps/Kartotherian|Kartotherian]] |
| | * [[Maps/Tilerator|Tilerator]] |
| | * [[Maps/Maintenance|Maintenance]] |
| | |
| | == Miscellaneous == |
| | * [https://wikitech.wikimedia.org/wiki/Maps/Dynamic_tile_sources Dynamic tile sources] |
| | * [[Maps/Debugging]] |
| | |
| | == Development processes == |
| | * [https://wikitech.wikimedia.org/wiki/Maps/Services_deployment Kartotherian/Tilerator deployment] |
| | |
| | == Puppetization and Automation == |
| | === Prerequisites === |
| | * passwords and postgres replication configuration is set in Ops private repo (<code>root@puppetmaster1001:/srv/private/hieradata/role/(codfw|eqiad)/maps/server.yaml</code>) |
| | * other configuration in <code>puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml</code> |
| | * <code>cassandra::rack</code> is defined in <code>puppet/hieradata/hosts/maps*.yaml</code> |
| | * the <code>role::maps::master</code> / <code>role::maps::slave</code> roles are associated to the maps nodes (site.pp) |
|
| |
|
| == Monitoring == | | == Monitoring == |
| Line 15: |
Line 40: |
| * [https://grafana.wikimedia.org/dashboard/db/service-tilerator Tilerator - Grafana] | | * [https://grafana.wikimedia.org/dashboard/db/service-tilerator Tilerator - Grafana] |
| * [https://logstash.wikimedia.org/#/dashboard/elasticsearch/tilerator Tilerator - Logstash] | | * [https://logstash.wikimedia.org/#/dashboard/elasticsearch/tilerator Tilerator - Logstash] |
| * [https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=cpu_report&s=by+name&c=Maps%2520caches%2520eqiad&tab=m&vn=&hide-hf=false Ganglia - Maps Varnish cluster] (eqiad)
| |
| * [https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&c=Maps+Cluster+codfw&tab=m&vn=&hide-hf=false&sh=1 Ganglia - Maps cluster] (codfw)
| |
| * [https://ganglia.wikimedia.org/latest/?r=day&cs=&ce=&c=Maps+Cluster+codfw&h=&tab=m&vn=&hide-hf=false&m=disk_free&sh=1&z=small&hc=4&host_regex=&max_graphs=0&s=by+name Free disk space]
| |
|
| |
| == Importing database ==
| |
| '''maps2001 is actually not the best server for this - we should switch it around with maps2002, as it has 12 cores and 96GB RAM.
| |
| * From http://planet.osm.org/pbf/ - find the file with the latest available date, but do NOT use "latest", as that might change at any moment.
| |
| * <tt><nowiki>curl -x webproxy.eqiad.wmnet:8080 -O http://planet.osm.org/pbf/planet-151214.osm.pbf.md5</nowiki></tt>
| |
| * <tt><nowiki>curl -x webproxy.eqiad.wmnet:8080 -O http://planet.osm.org/pbf/planet-151214.osm.pbf</nowiki></tt>
| |
| * <tt>md5sum -c planet-151214.osm.pbf.md5</tt>
| |
| * <tt>PGPASSWORD="$(< ~/osmimporter_pass)" osm2pgsql --create --slim --flat-nodes nodes.bin -C 40000 --number-processes 8 --hstore planet-151214.osm.pbf -H maps-test2001 -U osmimporter -d gis</tt>
| |
|
| |
| * additional steps to import shapes and create some indexes / functions / ... are documented in [https://github.com/kartotherian/osm-bright.tm2source#install Kartotherian sources].
| |
|
| |
| === Notes ===
| |
| * Tables are created by osm2pgsql, no need for an initial DDL script.
| |
|
| |
| == [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] ==
| |
| Kartotherian servers map tiles by getting vector data from Cassandra, applying the [https://github.com/kartotherian/osm-bright.tm2/blob/master/README.md style] to it, and returning raster images. It is also capable of serving a "static image" - a map with a given width/height/scaling/zoom, and can server vector tiles directly for on-the-client rendering (WebGL maps).
| |
| * [Https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/kartotherian/templates/config.yaml.erb Service configuration template] (extends [[phab:diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb|base template]])
| |
| * [https://phabricator.wikimedia.org/diffusion/GMKT/browse/master/sources.prod.yaml Sources configuration]
| |
| To see the tiles without Varnish cache, connect to Kartotherian using an ssh tunnel, e.g. <code>ssh -L 6533:localhost:6533 maps-test2001.codfw.wmnet</code> and browse to http://localhost:6533
| |
|
| |
| == [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] ==
| |
| Tilerator is a backend vector tile pre-generation service that picks up jobs from a Redis job que, copying tiles from a Postgres DB, using [https://github.com/kartotherian/osm-bright.tm2source/blob/master/README.md sql queries] into vector tiles stored in Cassandra. Postgres DBs are set up on each of the maps hosts, one master and 3 slaves. Technically, Tilerator is not even a generator, but rather a "batch copying" service, which takes tiles from one configured source (e.g. a tile generator from SQL), and puts it into another source (e.g. Cassandra tile store).
| |
| * [Https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/tilerator/templates/config.yaml.erb Service configuration template] (extends [[phab:diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb|base template]])
| |
| * [https://phabricator.wikimedia.org/diffusion/GMTL/browse/master/sources.prod.yaml Sources configuration]
| |
|
| |
| == TileratorUI ==
| |
| TileratorUI is used to add jobs to the Tilerator job queue. Actually, TileratorUI is the same code as Tilerator, but started with a different configuration. Connect to TileratorUI using an ssh tunnel, e.g. <code>ssh -L 6535:localhost:6535 maps-test2001.codfw.wmnet</code> and navigating to http://localhost:6535. There, you can view any style (use set style to change it), or to schedule a job by setting all relevant fields and Control+Clicking the tile you want to schedule.
| |
|
| |
| See full [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator documentation] for all commands & parameters.
| |
| === Dynamic Tile Sources ===
| |
|
| |
| ==== Cassandra ====
| |
| To create a new Cassandra data source, post something like this to the /sources as a text body. Default table name is <code>tiles</code>. If table or keyspace is not there, you have to use <code>createIfMissing</code> parameter.<syntaxhighlight lang="yaml">
| |
| v2a:
| |
| uri: cassandra://
| |
| params:
| |
| keyspace: v2
| |
| table: tiles2a
| |
| cp: [maps-test2001.codfw.wmnet, maps-test2002.codfw.wmnet, maps-test2003.codfw.wmnet, maps-test2004.codfw.wmnet]
| |
| username: {var: cassandra-user}
| |
| password: {var: cassandra-pswd}
| |
| # repfactor: 4
| |
| # durablewrite: 0
| |
| # createIfMissing: true
| |
| </syntaxhighlight>
| |
|
| |
| ==== Dynamic Layer Generator ====
| |
| To generate just a few layers from database, create a layer filter and a layer mixer:<syntaxhighlight lang="yaml">
| |
| gentmp:
| |
| uri: bridge://
| |
| xml:
| |
| npm: ["osm-bright-source", "data.xml"]
| |
| xmlSetDataSource:
| |
| if:
| |
| dbname: gis
| |
| host: ""
| |
| type: postgis
| |
| set:
| |
| host: localhost
| |
| user: {var: osmdb-user}
| |
| password: {var: osmdb-pswd}
| |
| xmlLayers: [admin, road]
| |
|
| |
| mixtmp:
| |
| uri: layermixer://
| |
| params:
| |
| sources: [{ref: v2}, {ref: gentmp}]
| |
| </syntaxhighlight>Once set, POST a job to copy <code>mixtmp</code> into the storage <code>v2</code> e.g.
| |
|
| |
| '''<code>src=mixtmp dst=v2 baseZoom=0 fromZoom=5 beforeZoom=6 parts=10</code>'''
| |
|
| |
| === Generating Tiles ===
| |
| Generate all tiles for zooms <code>0..7</code>, using generator <code>gen</code>, saving into <code>v3</code> everything including the solid tiles, up to 4 jobs per zoom.
| |
|
| |
| '''<code>src=gen dst=v3 parts=4 baseZoom=0 fromZoom=0 beforeZoom=8 saveSolid=1</code>'''
| |
|
| |
| Generated tiles only if they already exist in <code>v2</code> source, and save them into <code>v3</code>, on zooms <code>8..15</code>, 60 jobs per zoom.
| |
|
| |
| '''<code>src=gen dst=v3 parts=60 baseZoom=0 fromZoom=8 beforeZoom=16 sourceId=v2</code>'''
| |
|
| |
| === Bulk Copying ===
| |
| The fastest way to copy a large number of tiles from one source to another is to use a large number of parts and specify <code>saveSolid=true</code> (skips solid tile detection). E.g. to copy all z16 tiles from v2 to v3, use<blockquote>'''<code>src=v2 dst=v3 zoom=16 parts=60 saveSolid=true</code>''' </blockquote>
| |
|
| |
| == Postgres ==
| |
| * Clear the Postgres data directory and init the database from backup (replace <code>maps2001.codfw.wmnet</code> by the postgres master):
| |
| <code>rm -rf /srv/postgresql/9.4/main/* && sudo -u postgres pg_basebackup -X stream -D /srv/postgresql/9.4/main/ -h maps2001.codfw.wmnet -U replication -W</code>
| |
|
| |
| == Puppetization and Automation ==
| |
| === Prerequisites ===
| |
| * passwords and postgres replication configuration is set in Ops private repo (<code>root@palladium:~/private/hieradata/role/(codfw|eqiad)/maps/server.yaml</code>)
| |
| * other configuration in <code>puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml</code>
| |
| * <code>cassandra::rack</code> is defined in <code>puppet/hieradata/maps*.yaml</code>
| |
| * the <code>maps::server, maps::master</code> roles are associated to the maps master node (site.pp)
| |
| * the <code>maps::server, maps::slave</code> roles are associated to the maps slaves node (site.pp)
| |
|
| |
| === Manual steps ===
| |
|
| |
| * On first start of a Cassandra node, it is necessary to clean the data directory <code>rm -rf /srv/cassandra/*</code>. See [[Cassandra#Adding_a_new_.28empty.29_node|Cassandra documentation]] for more details.
| |
|
| |
| * To initialize the first Cassandra node, we need to add the local node to the list of seeds by manually editing <code>/etc/cassandra/cassandra.yaml</code> and restarting cassandra:
| |
|
| |
| <syntaxhighlight lang="yaml">
| |
| seed_provider:
| |
| # Addresses of hosts that are deemed contact points.
| |
| # Cassandra nodes use this list of hosts to find each other and learn
| |
| # the topology of the ring. You must change this if you are running
| |
| # multiple nodes!
| |
| - class_name: org.apache.cassandra.locator.SimpleSeedProvider
| |
| parameters:
| |
| # seeds is actually a comma-delimited list of addresses.
| |
| # Ex: "<ip1>,<ip2>,<ip3>"
| |
| # Omit own host name / IP in multi-node clusters (see
| |
| # https://phabricator.wikimedia.org/T91617).
| |
| - seeds: 10.64.16.42,10.64.32.117,10.64.48.154 # '''add local node here to initialize the first Cassandra node'''
| |
| </syntaxhighlight>
| |
|
| |
| * change the cassandra super user password to match the one configured in private repo using cqlsh:
| |
| cqlsh '''<maps1001.eqiad.wmnet>''' -u cassandra
| |
| ALTER USER cassandra WITH PASSWORD '<password>';
| |
| * Setup of user access / rights for cassandra
| |
| cat /usr/local/bin/maps-grants.cql | cqlsh '''<maps1001.eqiad.wmnet>''' -u cassandra
| |
| * Initial data load of OSM into postgresql is done by running <code>/usr/local/bin/osm-initial-import</code> on the postgresql master node.
| |
| <syntaxhighlight lang="bash">
| |
| osm-initial-import \
| |
| -d <date_of_import> \
| |
| -p <password_file> \
| |
| -s <state_file_url> \
| |
| -x webproxy.eqiad.wmnet:8080
| |
| </syntaxhighlight>
| |
|
| |
| ** '''date_of_import:''' find the latest dump at https://planet.osm.org/pbf/. Example: <code>160530</code>.
| |
| ** '''password_file:''' A file containing the postgresql password of the osmimporter user.
| |
| ** '''state_file_url:''' The URL to the state file corresponding to the dump, find the correct one at http://planet.openstreetmap.org/replication/ (the state file must be older than the dump). Example: <code>http://planet.openstreetmap.org/replication/day/000/001/355.state.txt</code>.
| |
|
| |
| * If the postgresql master already has data, the slave initialization will timeout in puppet. It then needs to be run manually:
| |
|
| |
| <syntaxhighlight lang="bash">
| |
| service postgresql@9.4-main stop
| |
|
| |
| rm -rf /srv/postgresql/9.4/main
| |
| mkdir /srv/postgresql/9.4/main
| |
| chown postgres: /srv/postgresql/9.4/main/
| |
| chmod 700 /srv/postgresql/9.4/main/
| |
|
| |
| sudo -u postgres pg_basebackup \
| |
| --xlog-method=stream \
| |
| --pgdata=/srv/postgresql/9.4/main/ \
| |
| --host='''<postgres_master>''' \
| |
| --username=replication \
| |
| --write-recovery-conf \
| |
| --password
| |
|
| |
| service postgresql@9.4-main start
| |
| </syntaxhighlight>
| |
|
| |
|
| |
| * Initial creation of cassandra keyspace: To prevent accidental modification of schema, Tilerator source configuration does not allow to create schema by default. The sources file used by tilerator / kartotherian is configured in <code>/etc/(kartotherian|tilerator|tileratorui)/config.yaml</code>, look for the <code>sources:</code> key. This is a reference to a sources file in the kartotherian / tilerator source directory. For example <code>/srv/deployment/tilerator/deploy/src/sources.prod.yaml</code>.
| |
|
| |
| The easiest way to create a new keyspace is to run Tilerator with a custom sources file, which instructs tilerator to create the missing keyspace. For example, create a temporary file, e.g. <code>/home/yurik/my-source-file</code> with the following configuration:
| |
| <syntaxhighlight lang="yaml">
| |
| v5:
| |
| uri: cassandra://
| |
| params:
| |
| keyspace: v5
| |
| cp: {var: cassandra-servers}
| |
| username: {var: cassandra-user}
| |
| password: {var: cassandra-pswd}
| |
| # These parameters are only used if keyspace needs to be created:
| |
| repfactor: 4
| |
| durablewrite: 0
| |
| createIfMissing: true
| |
| </syntaxhighlight>
| |
|
| |
| And run this bash script:
| |
| <syntaxhighlight lang="bash">
| |
| node /srv/deployment/tilerator/deploy/src/scripts/tileshell.js \
| |
| # Use TileratorUI configuration (including password variables)
| |
| --config /etc/tileratorui/config.yaml \
| |
| # Use this sources file instead of the one in the config file
| |
| --source /home/yurik/my-source-file
| |
| </syntaxhighlight>
| |
|
| |
| * On existing server, record all existing tiles as a list of tile indexes
| |
| <syntaxhighlight lang="bash">
| |
| node /srv/deployment/tilerator/deploy/src/scripts/tileshell.js \
| |
| --config /etc/tileratorui/config.yaml \
| |
| # List all tiles in the "v5" source
| |
| -j.generatorId v5 \
| |
| # Which zoom to enumerate
| |
| -j.zoom 14 \
| |
| # File to write indexes to
| |
| --dumptiles /home/yurik/all-tiles-14.txt \
| |
| # Instead of dumping indexes in "zoom/x/y" format, write one number indexes (0..4^zoom-1)
| |
| --dumprawidx \
| |
| # If dumptiles file already exists, override it
| |
| --dumpoverride
| |
| </syntaxhighlight>
| |
| * Instead of generating the entire zoom level, you may want to generate just the tiles in a list
| |
| <syntaxhighlight lang="bash">
| |
| node /srv/deployment/tilerator/deploy/src/scripts/tileshell.js \
| |
| --config /etc/tileratorui/config.yaml \
| |
| # List of tile indexes, unique and sorted, one per line.
| |
| # Indexes can be 0..4^zoom-1
| |
| -j.filepath /home/yurik/all-tiles-14.txt \
| |
| # All tile indexes in the file belong to zoom 14
| |
| # Without this parameter, the file must contain zoom/x/y triplets
| |
| -j.fileZoomOverride 14 \
| |
| # generate zoom levels 10 <= zoom < 16
| |
| -j.fromZoom 10 -j.beforeZoom 16 \
| |
| # Copy tiles from "gen" source to "v3" source
| |
| -j.generatorId gen -j.storageId v3 \
| |
| # If tile already exists in "v3", but "gen" produces an empty tile, delete it in v3
| |
| -j.deleteEmpty
| |
| </syntaxhighlight>
| |
|
| |
| == Building Kartotherian and Tilerator ==
| |
| While Kartotherian uses the same build system as [https://github.com/wikimedia/service-template-node Service-template-node], it also needs to use npm shrinkwrap to prevent migration from Mapnik 3.5.13 to 3.5.14 due to the updated CLIB dependency. After your system is configured for the regular service-template-node build, you need to do these annoying steps to actually build it (identical for both Kartotherian and Tilerator):
| |
| <syntaxhighlight lang="bash">
| |
| # Build Kartotherian using Docker container
| |
| kartotherian$ ./server.js build --deploy-repo --force
| |
|
| |
| kartotherian$ cd ../deploy
| |
| deploy$ npm update
| |
| deploy$ npm shrinkwrap
| |
| deploy$ mv npm-shrinkwrap.json ..
| |
| deploy$ git clean -f -d
| |
| deploy$ git checkout master
| |
| deploy$ git submodule update
| |
| deploy$ mv ../npm-shrinkwrap.json .
| |
| # Compare npm-shrinkwrap.json to the previous version, revert Mapnik's update back to 3.5.13
| |
| deploy$ meld .
| |
| deploy$ git add npm-shrinkwrap.json
| |
| deploy$ git commit
| |
| deploy$ git push # or git review and +2 via gerrit
| |
| deploy$ cd ../kartotehrian
| |
| # During the build Docker container will now use the latest npm-shrinkwrap
| |
| kartotherian$ ./server.js build --deploy-repo --force
| |
| kartotherian$ cd ../deploy
| |
| deploy$ git review
| |
|
| |
|
| </syntaxhighlight>
| | == Subpages == |
| | {{Special:PrefixIndex/{{PAGENAME}}/|hideredirects=1|stripprefix=1}} |
