imported>Quiddity |
imported>Effie Mouzeli |
| (9 intermediate revisions by 4 users not shown) |
| Line 1: |
Line 1: |
| This page describes the technical aspects of deploying [[mw:Maps|Maps service]] on Wikimedia Foundation infrastructure. | | This page describes the technical aspects of deploying [[mw:Maps|Maps service]] on Wikimedia Foundation infrastructure. |
| | |
| | '''The service is being actively redesigned, documentation can me found under [[Maps/v2]]''' |
|
| |
|
| == Intro == | | == Intro == |
| [[File:Maps-components.png|thumb|Maps service component diagram]] | | [[File:Maps-components.png|thumb|Maps service component diagram]] |
| [[File:Maps-deployment.png|thumb|Maps service deployment diagram]] | | [[File:Maps-deployment.png|thumb|Maps service deployment diagram]] |
| The maps service consists of [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] - a nodejs service to serve map tiles, [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are four servers in the <code>maps</code> group: <code>maps-test200{1,2,3,4}.codfw.wmnet</code> that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are four Varnish servers per datacenter in the <code>cache_maps</code> group.
| | [[File:Maps_@_PI@2x.png|thumb|Kartotherian internals focus]] |
| | |
| == Monitoring ==
| |
| * [https://grafana.wikimedia.org/dashboard/db/interactive-team-kpi KPI dashboard]
| |
| * [http://discovery.wmflabs.org/maps/ Usage dashboard]
| |
| * [https://grafana.wikimedia.org/dashboard/db/service-maps-varnish Usage - varnish]
| |
| * [https://grafana.wikimedia.org/dashboard/db/service-kartotherian Kartotherian - Grafana]
| |
| * [https://logstash.wikimedia.org/#/dashboard/elasticsearch/kartotherian Kartotherian - Logstash]
| |
| * [https://logstash.wikimedia.org/#/dashboard/elasticsearch/maps-cassandra Maps Cassandra - Logstash]
| |
| * [https://grafana.wikimedia.org/dashboard/db/service-tilerator Tilerator - Grafana]
| |
| * [https://logstash.wikimedia.org/#/dashboard/elasticsearch/tilerator Tilerator - Logstash]
| |
| | |
| == Importing database ==
| |
| '''maps2001 is actually not the best server for this - we should switch it around with maps2002, as it has 12 cores and 96GB RAM.
| |
| * From https://planet.openstreetmap.org/pbf/ - find the file with the latest available date, but do NOT use "latest", as that might change at any moment.
| |
| * <tt><nowiki>curl -x webproxy.eqiad.wmnet:8080 -O https://planet.openstreetmap.org/pbf/planet-151214.osm.pbf.md5</nowiki></tt>
| |
| * <tt><nowiki>curl -x webproxy.eqiad.wmnet:8080 -O https://planet.openstreetmap.org/pbf/planet-151214.osm.pbf</nowiki></tt>
| |
| * <tt>md5sum -c planet-151214.osm.pbf.md5</tt>
| |
| * <tt>sudo -u osmupdater PGPASSWORD="$(< ~/osmimporter_pass)" osm2pgsql --create --slim --flat-nodes nodes.bin -C 40000 --number-processes 8 --hstore planet-151214.osm.pbf -H localhost -U osmupdater -d gis</tt>
| |
| | |
| * additional steps to import shapes and create some indexes / functions / ... are documented in [https://github.com/kartotherian/osm-bright.tm2source#install Kartotherian sources].
| |
| ** Note: after importing the water polygons, permissions will need to be granted once again to allow users to read water_polygons: <tt><nowiki>sudo -u postgres psql -d gis -c 'GRANT SELECT ON ALL TABLES IN SCHEMA public TO kartotherian, tilerator, tileratorui;'</nowiki></tt>
| |
| | |
| === Notes ===
| |
| * Tables are created by osm2pgsql, no need for an initial DDL script.
| |
| | |
| == [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] ==
| |
| Kartotherian serves map tiles by getting vector data from Cassandra, applying the [https://github.com/kartotherian/osm-bright.tm2/blob/master/README.md style] to it, and returning raster images. It is also capable of serving a "static image" - a map with a given width/height/scaling/zoom, and can server vector tiles directly for on-the-client rendering (WebGL maps).
| |
| * [Https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/kartotherian/templates/config.yaml.erb Service configuration template] (extends [[phab:diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb|base template]])
| |
| * [https://phabricator.wikimedia.org/diffusion/GMKT/browse/master/sources.prod.yaml Sources configuration]
| |
| To see the tiles without Varnish cache, connect to Kartotherian using an ssh tunnel, e.g. <code>ssh -L 6533:localhost:6533 maps-test2001.codfw.wmnet</code> and browse to http://localhost:6533
| |
|
| |
|
| == [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] ==
| | The maps service consists of [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] - a nodejs service to serve map tiles, [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are 20 servers in the <code>maps</code> group: <code>maps20[01-10].codfw.wmnet</code> and <code>maps10[01-10].eqiad.wmnet</code> that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are four Varnish servers per datacenter in the <code>cache_maps</code> group. |
| Tilerator is a backend vector tile pre-generation service that picks up jobs from a Redis job que, copying tiles from a Postgres DB, using [https://github.com/kartotherian/osm-bright.tm2source/blob/master/README.md sql queries] into vector tiles stored in Cassandra. Postgres DBs are set up on each of the maps hosts, one master and 3 slaves. Technically, Tilerator is not even a generator, but rather a "batch copying" service, which takes tiles from one configured source (e.g. a tile generator from SQL), and puts it into another source (e.g. Cassandra tile store).
| |
| * [Https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/tilerator/templates/config.yaml.erb Service configuration template] (extends [[phab:diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb|base template]])
| |
| * [https://phabricator.wikimedia.org/diffusion/GMTL/browse/master/sources.prod.yaml Sources configuration]
| |
|
| |
|
| == TileratorUI == | | == The infrastructure == |
| TileratorUI is used to inspect maps, including internal data sources, and to add jobs to the Tilerator job queue. Actually, TileratorUI is the same code as Tilerator, but started with a different configuration. Connect to TileratorUI using an ssh tunnel, e.g. <code>ssh -L 6535:localhost:6535 maps-test2001.codfw.wmnet</code> and navigating to http://localhost:6535. There, you can view any style (use set style to change it), or to schedule a job by setting all relevant fields and Control+Clicking the tile you want to schedule.
| | * [[Maps/OSM Database|OSM Database]] |
| | * [[Maps/Tile_storage|Tile storage]] |
| | * [[Maps/Kartotherian|Kartotherian]] |
| | * [[Maps/Tilerator|Tilerator]] |
| | * [[Maps/Maintenance|Maintenance]] |
|
| |
|
| [[File:Tilerator UI.png|thumb|center|500px|Screenshot]] | | == Miscellaneous == |
| | * [https://wikitech.wikimedia.org/wiki/Maps/Dynamic_tile_sources Dynamic tile sources] |
| | * [[Maps/Debugging]] |
|
| |
|
| === Quick cheat sheet === | | == Development processes == |
| * Style is specified in upper left corner | | * [https://wikitech.wikimedia.org/wiki/Maps/Services_deployment Kartotherian/Tilerator deployment] |
| ** Set it to <code>genview</code> to view tiles generated on the fly. '''Caution''': that means that if you zoom out to low levels tiles can take more that 10 minutes to generate.
| |
| * Alt+click (Option+click on Mac) on map to enqueue regeneration jobs.
| |
| ** This requires <code>src</code> and <code>dst</code> to be set. For the most basic operation, on-demand regeneration of tiles, set <code>src</code> to <code>gen</code> and <code>dst</code> to whatever Cassandra keyspace is used for tile storage (currently <code>v4</code>).
| |
| ** By default, only tile clicked upon will be regenerated.
| |
| ** Set <code>fromZ</code> and <code>beforeZ</code> to regenerate a bunch of layers under the clicked tile.
| |
| * Click on <code>source</code> to view the currently active sources configuration.
| |
| | |
| See full [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator documentation] for all commands & parameters.
| |
| | |
| === Dynamic tile sources ===
| |
| | |
| ==== Cassandra ====
| |
| To create a new Cassandra data source, post something like this to the /sources as a text body. Default table name is <code>tiles</code>. If table or keyspace is not there, you have to use <code>createIfMissing</code> parameter.<syntaxhighlight lang="yaml">
| |
| v2a:
| |
| uri: cassandra://
| |
| params:
| |
| keyspace: v2
| |
| table: tiles2a
| |
| cp: [maps-test2001.codfw.wmnet, maps-test2002.codfw.wmnet, maps-test2003.codfw.wmnet, maps-test2004.codfw.wmnet]
| |
| username: {var: cassandra-user}
| |
| password: {var: cassandra-pswd}
| |
| # repfactor: 4
| |
| # durablewrite: 0
| |
| # createIfMissing: true
| |
| # copyInfoFrom: {ref: gen}
| |
| </syntaxhighlight>
| |
| | |
| ==== Dynamic Layer Generator ====
| |
| To generate just a few layers from database, create a layer filter and a layer mixer:<syntaxhighlight lang="yaml">
| |
| gentmp:
| |
| uri: bridge://
| |
| xml:
| |
| npm: ["osm-bright-source", "data.xml"]
| |
| xmlSetDataSource:
| |
| if:
| |
| dbname: gis
| |
| host: ""
| |
| type: postgis
| |
| set:
| |
| host: localhost
| |
| user: {var: osmdb-user}
| |
| password: {var: osmdb-pswd}
| |
| xmlLayers: [admin, road]
| |
| | |
| mixtmp:
| |
| uri: layermixer://
| |
| params:
| |
| sources: [{ref: v2}, {ref: gentmp}]
| |
| </syntaxhighlight>Once set, POST a job to copy <code>mixtmp</code> into the storage <code>v2</code> e.g.
| |
| | |
| '''<code>src=mixtmp dst=v2 baseZoom=0 fromZoom=5 beforeZoom=6 parts=10</code>'''
| |
| | |
| === Generating Tiles ===
| |
| Generate all tiles for zooms <code>0..7</code>, using generator <code>gen</code>, saving into <code>v4</code> everything including the solid tiles, up to 4 jobs per zoom.
| |
| | |
| '''<code>src=gen dst=v4 parts=4 baseZoom=0 fromZoom=0 beforeZoom=8 saveSolid=1</code>'''
| |
| | |
| Generated tiles only if they already exist in <code>v2</code> source, and save them into <code>v4</code>, on zooms <code>8..15</code>, 60 jobs per zoom.
| |
| | |
| '''<code>src=gen dst=v4 parts=60 baseZoom=0 fromZoom=8 beforeZoom=16 sourceId=v2</code>'''
| |
| | |
| === Bulk Copying ===
| |
| The fastest way to copy a large number of tiles from one source to another is to use a large number of parts and specify <code>saveSolid=true</code> (skips solid tile detection). E.g. to copy all z16 tiles from v3 to v4, use '''<code>src=v3 dst=v4 zoom=16 parts=60 saveSolid=true</code>'''
| |
| | |
| == Postgres ==
| |
| * Clear the [[Postgres]] data directory and init the database from backup (replace <code>maps2001.codfw.wmnet</code> by the postgres master):
| |
| <code>rm -rf /srv/postgresql/9.4/main/* && sudo -u postgres pg_basebackup -X stream -D /srv/postgresql/9.4/main/ -h maps2001.codfw.wmnet -U replication -W</code>
| |
|
| |
|
| == Puppetization and Automation == | | == Puppetization and Automation == |
| === Prerequisites === | | === Prerequisites === |
| * passwords and postgres replication configuration is set in Ops private repo (<code>root@palladium:~/private/hieradata/role/(codfw|eqiad)/maps/server.yaml</code>) | | * passwords and postgres replication configuration is set in Ops private repo (<code>root@puppetmaster1001:/srv/private/hieradata/role/(codfw|eqiad)/maps/server.yaml</code>) |
| * other configuration in <code>puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml</code> | | * other configuration in <code>puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml</code> |
| * <code>cassandra::rack</code> is defined in <code>puppet/hieradata/hosts/maps*.yaml</code> | | * <code>cassandra::rack</code> is defined in <code>puppet/hieradata/hosts/maps*.yaml</code> |
| * the <code>role::maps::master</code> / <code>role::maps::slave</code> roles are associated to the maps nodes (site.pp) | | * the <code>role::maps::master</code> / <code>role::maps::slave</code> roles are associated to the maps nodes (site.pp) |
|
| |
|
| === Manual steps === | | == Monitoring == |
| | | * [https://grafana.wikimedia.org/dashboard/db/interactive-team-kpi KPI dashboard] |
| * On first start of a Cassandra node, it is necessary to clean the data directory <code>rm -rf /srv/cassandra/*</code>. See [[Cassandra#Adding_a_new_.28empty.29_node|Cassandra documentation]] for more details. | | * [http://discovery.wmflabs.org/maps/ Usage dashboard] |
| | | * [https://grafana.wikimedia.org/dashboard/db/service-maps-varnish Usage - varnish] |
| * To initialize the first Cassandra node, we need to add the local node to the list of seeds by manually editing <code>/etc/cassandra/cassandra.yaml</code> and restarting cassandra:
| | * [https://grafana.wikimedia.org/dashboard/db/service-kartotherian Kartotherian - Grafana] |
| | | * [https://logstash.wikimedia.org/#/dashboard/elasticsearch/kartotherian Kartotherian - Logstash] |
| <syntaxhighlight lang="yaml">
| | * [https://logstash.wikimedia.org/#/dashboard/elasticsearch/maps-cassandra Maps Cassandra - Logstash] |
| seed_provider:
| | * [https://grafana.wikimedia.org/dashboard/db/service-tilerator Tilerator - Grafana] |
| # Addresses of hosts that are deemed contact points.
| | * [https://logstash.wikimedia.org/#/dashboard/elasticsearch/tilerator Tilerator - Logstash] |
| # Cassandra nodes use this list of hosts to find each other and learn
| |
| # the topology of the ring. You must change this if you are running
| |
| # multiple nodes!
| |
| - class_name: org.apache.cassandra.locator.SimpleSeedProvider
| |
| parameters:
| |
| # seeds is actually a comma-delimited list of addresses.
| |
| # Ex: "<ip1>,<ip2>,<ip3>"
| |
| # Omit own host name / IP in multi-node clusters (see
| |
| # https://phabricator.wikimedia.org/T91617).
| |
| - seeds: 10.64.16.42,10.64.32.117,10.64.48.154 # '''add local node here to initialize the first Cassandra node'''
| |
| </syntaxhighlight>
| |
| | |
| * change the cassandra super user password to match the one configured in private repo using cqlsh:
| |
| cqlsh '''<maps1001.eqiad.wmnet>''' -u cassandra
| |
| Password: cassandra
| |
| ALTER USER cassandra WITH PASSWORD '<password>';
| |
| * Setup of user access / rights for cassandra
| |
| cat /usr/local/bin/maps-grants.cql | cqlsh '''<maps1001.eqiad.wmnet>''' -u cassandra
| |
| * Setup replication of Cassandra <code>system_auth</code> according to [[Cassandra#Replicating_system_auth|documentation]]. | |
| * Initial data load of OSM into postgresql is done by running <code>/usr/local/bin/osm-initial-import</code> on the postgresql master node. Cassandra should be shutdown during initial import to free memory for osm2pgsql.
| |
| <syntaxhighlight lang="bash">
| |
| osm-initial-import \
| |
| -d <date_of_import> \
| |
| -p <password_file> \
| |
| -s <state_file_url> \
| |
| -x webproxy.eqiad.wmnet:8080
| |
| </syntaxhighlight>
| |
| | |
| ** '''date_of_import:''' find the latest dump at https://planet.osm.org/pbf/. Example: <code>160530</code>.
| |
| ** '''password_file:''' A file containing the postgresql password of the osmimporter user. | |
| ** '''state_file_url:''' The URL to the state file corresponding to the dump, find the correct one at http://planet.openstreetmap.org/replication/ (the state file must be older than the dump). Example: <code>http://planet.openstreetmap.org/replication/day/000/001/355.state.txt</code>.
| |
| | |
| * If the postgresql master already has data, the slave initialization will timeout in puppet. It then needs to be run manually:
| |
| | |
| <syntaxhighlight lang="bash">
| |
| service postgresql@9.4-main stop | |
| | |
| rm -rf /srv/postgresql/9.4/main
| |
| mkdir /srv/postgresql/9.4/main
| |
| chown postgres: /srv/postgresql/9.4/main/
| |
| chmod 700 /srv/postgresql/9.4/main/
| |
| | |
| sudo -u postgres pg_basebackup \
| |
| --xlog-method=stream \
| |
| --pgdata=/srv/postgresql/9.4/main/ \
| |
| --host='''<postgres_master>''' \
| |
| --username=replication \
| |
| --write-recovery-conf \
| |
| --password \
| |
| --progress \
| |
| --verbose
| |
| | |
| service postgresql@9.4-main start | |
| </syntaxhighlight>
| |
| | |
| | |
| * Initial creation of cassandra keyspace: To prevent accidental modification of schema, Tilerator source configuration does not allow to create schema by default. The sources file used by tilerator / kartotherian is configured in <code>/etc/(kartotherian|tilerator|tileratorui)/config.yaml</code>, look for the <code>sources:</code> key. This is a reference to a sources file in the kartotherian / tilerator source directory. For example <code>/srv/deployment/tilerator/deploy/src/sources.prod2.yaml</code>.
| |
| | |
| The easiest way to create a new keyspace is to run Tilerator with a custom sources file, which instructs tilerator to create the missing keyspace. For example, create a temporary file, e.g. <code>/home/yurik/my-source-file</code> with the following configuration (change <code>v4</code> with the keyspace declared in the sources configuration file):
| |
| <syntaxhighlight lang="yaml">
| |
| v4:
| |
| uri: cassandra://
| |
| params:
| |
| keyspace: v4
| |
| cp: {var: cassandra-servers}
| |
| username: {var: cassandra-user}
| |
| password: {var: cassandra-pswd}
| |
| # These parameters are only used if keyspace needs to be created:
| |
| repfactor: 4
| |
| durablewrite: 0
| |
| createIfMissing: true
| |
| copyInfoFrom: {ref: gen}
| |
| </syntaxhighlight>
| |
| | |
| And run this bash script:
| |
| <syntaxhighlight lang="bash">
| |
| node /srv/deployment/tilerator/deploy/node_modules/tilerator/scripts/tileshell.js \
| |
| # Use TileratorUI configuration (including password variables)
| |
| --config /etc/tileratorui/config.yaml \
| |
| # Use this sources file instead of the one in the config file
| |
| --source /home/yurik/my-source-file
| |
| </syntaxhighlight>
| |
| | |
| Tileshell will not exit, so ^C it after it reports "done".
| |
| | |
| * On existing server, record all existing tiles as a list of tile indexes (path and generatorId need to be adapted) | |
| <syntaxhighlight lang="bash">
| |
| node /srv/deployment/tilerator/deploy/node_modules/tilerator/scripts/tileshell.js \
| |
| --config /etc/tileratorui/config.yaml \
| |
| # List all tiles in the "v5" source
| |
| -j.generatorId v5 \
| |
| # Which zoom to enumerate
| |
| -j.zoom 14 \
| |
| # File to write indexes to
| |
| --dumptiles /home/yurik/all-tiles-14.txt \
| |
| # Instead of dumping indexes in "zoom/x/y" format, write one number indexes (0..4^zoom-1)
| |
| --dumprawidx \
| |
| # If dumptiles file already exists, override it
| |
| --dumpoverride
| |
| </syntaxhighlight>
| |
| * Instead of generating the entire zoom level, you may want to generate just the tiles in a list (all parameters might need to be adapted)
| |
| <syntaxhighlight lang="bash">
| |
| node /srv/deployment/tilerator/deploy/node_modules/tilerator/scripts/tileshell.js \
| |
| --config /etc/tileratorui/config.yaml \
| |
| # List of tile indexes, unique and sorted, one per line.
| |
| # Indexes can be 0..4^zoom-1
| |
| -j.filepath /home/yurik/all-tiles-14.txt \
| |
| # All tile indexes in the file belong to zoom 14
| |
| # Without this parameter, the file must contain zoom/x/y triplets
| |
| -j.fileZoomOverride 14 \
| |
| # generate zoom levels 10 <= zoom < 16
| |
| -j.fromZoom 10 -j.beforeZoom 16 \
| |
| # Copy tiles from "gen" source to "v4" source
| |
| -j.generatorId gen -j.storageId v4 \
| |
| # If tile already exists in "v4", but "gen" produces an empty tile, delete it in v4
| |
| -j.deleteEmpty
| |
| </syntaxhighlight>
| |
| | |
| == Deploying maps services ==
| |
| | |
| === Git repositories galore ===
| |
| Kartotherian is developed [https://github.com/kartotherian on GitHub], however for deployment purposes we have copies of both services' main repos in Gerrit, at <code>[https://gerrit.wikimedia.org/r/#/admin/projects/maps/kartotherian maps/kartotherian]</code> and <code>[https://gerrit.wikimedia.org/r/#/admin/projects/maps/tilerator maps/tilerator]</code>. All development should happen on GitHub while Gerrit can have WMF-specific hacks (that should be avoided, however).
| |
| | |
| ==== Keeping repositories in sync ====
| |
| Assuming you have a checkout from Gerrit, create a remote called <code>github</code>:
| |
| | |
| kartotherian$ git remote add github https://github.com/kartotherian/kartotherian.git
| |
| | |
| Then when you need to sync the repositories, pull from GitHub and push to Gerrit:
| |
| | |
| git pull github master
| |
| git push
| |
| | |
| === Building for deployment ===
| |
| Kartotherian and Tilerator are deployed according to the [[mw:ServiceTemplateNode/Deployment|standard process for deploying]] Wikimedia Node.js services, with the important difference that deployments are built from purpose-specific "package" repos rather than directly from service code repos. This is to facilitate the bundling of additional maps-specific dependencies.
| |
| | |
| Kartotherian: https://gerrit.wikimedia.org/r/#/admin/projects/maps/kartotherian/package<br>
| |
| Tilerator: https://gerrit.wikimedia.org/r/#/admin/projects/maps/tilerator/package
| |
| | |
| === Deploying services ===
| |
| Refer to [[Services/Deployment]] for general instructions.
| |
| | |
| In most cases, Kartotherian and Tilerator should be deployed together, to ensure that all dependencies (in particular, styles) are in sync.
| |
|
| |
|
| == Subpages == | | == Subpages == |
| {{Special:PrefixIndex/{{PAGENAME}}/|hideredirects=1|stripprefix=1}} | | {{Special:PrefixIndex/{{PAGENAME}}/|hideredirects=1|stripprefix=1}} |
