imported>MSantos |
imported>Effie Mouzeli |
| (7 intermediate revisions by 3 users not shown) |
| Line 1: |
Line 1: |
| This page describes the technical aspects of deploying [[mw:Maps|Maps service]] on Wikimedia Foundation infrastructure. | | This page describes the technical aspects of deploying [[mw:Maps|Maps service]] on Wikimedia Foundation infrastructure. |
| | |
| | '''The service is being actively redesigned, documentation can me found under [[Maps/v2]]''' |
|
| |
|
| == Intro == | | == Intro == |
| [[File:Maps-components.png|thumb|Maps service component diagram]] | | [[File:Maps-components.png|thumb|Maps service component diagram]] |
| [[File:Maps-deployment.png|thumb|Maps service deployment diagram]] | | [[File:Maps-deployment.png|thumb|Maps service deployment diagram]] |
| The maps service consists of [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] - a nodejs service to serve map tiles, [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are four servers in the <code>maps</code> group: <code>maps-test200{1,2,3,4}.codfw.wmnet</code> that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are four Varnish servers per datacenter in the <code>cache_maps</code> group. | | [[File:Maps_@_PI@2x.png|thumb|Kartotherian internals focus]] |
| | |
| | The maps service consists of [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] - a nodejs service to serve map tiles, [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are 20 servers in the <code>maps</code> group: <code>maps20[01-10].codfw.wmnet</code> and <code>maps10[01-10].eqiad.wmnet</code> that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are four Varnish servers per datacenter in the <code>cache_maps</code> group. |
|
| |
|
| == The infrastructure == | | == The infrastructure == |
| * [https://wikitech.wikimedia.org/wiki/Maps/OSM_Database OSM Database (PostgreSQL)] | | * [[Maps/OSM Database|OSM Database]] |
| * Tile storage (Cassandra) | | * [[Maps/Tile_storage|Tile storage]] |
| * Kartotherian | | * [[Maps/Kartotherian|Kartotherian]] |
| * Tilerator | | * [[Maps/Tilerator|Tilerator]] |
| | * [[Maps/Maintenance|Maintenance]] |
| | |
| | == Miscellaneous == |
| | * [https://wikitech.wikimedia.org/wiki/Maps/Dynamic_tile_sources Dynamic tile sources] |
| | * [[Maps/Debugging]] |
| | |
| | == Development processes == |
| | * [https://wikitech.wikimedia.org/wiki/Maps/Services_deployment Kartotherian/Tilerator deployment] |
| | |
| | == Puppetization and Automation == |
| | === Prerequisites === |
| | * passwords and postgres replication configuration is set in Ops private repo (<code>root@puppetmaster1001:/srv/private/hieradata/role/(codfw|eqiad)/maps/server.yaml</code>) |
| | * other configuration in <code>puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml</code> |
| | * <code>cassandra::rack</code> is defined in <code>puppet/hieradata/hosts/maps*.yaml</code> |
| | * the <code>role::maps::master</code> / <code>role::maps::slave</code> roles are associated to the maps nodes (site.pp) |
|
| |
|
| == Monitoring == | | == Monitoring == |
| Line 21: |
Line 40: |
| * [https://grafana.wikimedia.org/dashboard/db/service-tilerator Tilerator - Grafana] | | * [https://grafana.wikimedia.org/dashboard/db/service-tilerator Tilerator - Grafana] |
| * [https://logstash.wikimedia.org/#/dashboard/elasticsearch/tilerator Tilerator - Logstash] | | * [https://logstash.wikimedia.org/#/dashboard/elasticsearch/tilerator Tilerator - Logstash] |
|
| |
| == [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] ==
| |
| Kartotherian serves map tiles by getting vector data from Cassandra, applying the [https://github.com/kartotherian/osm-bright.tm2/blob/master/README.md style] to it, and returning raster images. It is also capable of serving a "static image" - a map with a given width/height/scaling/zoom, and can server vector tiles directly for on-the-client rendering (WebGL maps).
| |
| * [Https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/kartotherian/templates/config.yaml.erb Service configuration template] (extends [[phab:diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb|base template]])
| |
| * [https://phabricator.wikimedia.org/diffusion/GMKT/browse/master/sources.prod.yaml Sources configuration]
| |
| To see the tiles without Varnish cache, connect to Kartotherian using an ssh tunnel, e.g. <code>ssh -L 6533:localhost:6533 maps-test2001.codfw.wmnet</code> and browse to http://localhost:6533
| |
|
| |
| == [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator] ==
| |
| Tilerator is a backend vector tile pre-generation service that picks up jobs from a Redis job que, copying tiles from a Postgres DB, using [https://github.com/kartotherian/osm-bright.tm2source/blob/master/README.md sql queries] into vector tiles stored in Cassandra. Postgres DBs are set up on each of the maps hosts, one master and 3 slaves. Technically, Tilerator is not even a generator, but rather a "batch copying" service, which takes tiles from one configured source (e.g. a tile generator from SQL), and puts it into another source (e.g. Cassandra tile store).
| |
| * [Https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/tilerator/templates/config.yaml.erb Service configuration template] (extends [[phab:diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb|base template]])
| |
| * [https://phabricator.wikimedia.org/diffusion/GMTL/browse/master/sources.prod.yaml Sources configuration]
| |
|
| |
| == TileratorUI ==
| |
| TileratorUI is used to inspect maps, including internal data sources, and to add jobs to the Tilerator job queue. Actually, TileratorUI is the same code as Tilerator, but started with a different configuration. Connect to TileratorUI using an ssh tunnel, e.g. <code>ssh -L 6535:localhost:6535 maps-test2001.codfw.wmnet</code> and navigating to http://localhost:6535. There, you can view any style (use set style to change it), or to schedule a job by setting all relevant fields and Control+Clicking the tile you want to schedule.
| |
|
| |
| [[File:Tilerator UI.png|thumb|center|500px|Screenshot]]
| |
|
| |
| === Quick cheat sheet ===
| |
| * Style is specified in upper left corner
| |
| ** Set it to <code>genview</code> to view tiles generated on the fly. '''Caution''': that means that if you zoom out to low levels tiles can take more that 10 minutes to generate.
| |
| * Alt+click (Option+click on Mac) on map to enqueue regeneration jobs.
| |
| ** This requires <code>src</code> and <code>dst</code> to be set. For the most basic operation, on-demand regeneration of tiles, set <code>src</code> to <code>gen</code> and <code>dst</code> to whatever Cassandra keyspace is used for tile storage (currently <code>v4</code>).
| |
| ** By default, only tile clicked upon will be regenerated.
| |
| ** Set <code>fromZ</code> and <code>beforeZ</code> to regenerate a bunch of layers under the clicked tile.
| |
| * Click on <code>source</code> to view the currently active sources configuration.
| |
|
| |
| See full [https://github.com/kartotherian/tilerator/blob/master/README.md Tilerator documentation] for all commands & parameters.
| |
|
| |
| === Dynamic tile sources ===
| |
|
| |
| ==== Cassandra ====
| |
| To create a new Cassandra data source, post something like this to the /sources as a text body. Default table name is <code>tiles</code>. If table or keyspace is not there, you have to use <code>createIfMissing</code> parameter.<syntaxhighlight lang="yaml">
| |
| v2a:
| |
| uri: cassandra://
| |
| params:
| |
| keyspace: v2
| |
| table: tiles2a
| |
| cp: [maps-test2001.codfw.wmnet, maps-test2002.codfw.wmnet, maps-test2003.codfw.wmnet, maps-test2004.codfw.wmnet]
| |
| username: {var: cassandra-user}
| |
| password: {var: cassandra-pswd}
| |
| # repfactor: 4
| |
| # durablewrite: 0
| |
| # createIfMissing: true
| |
| # copyInfoFrom: {ref: gen}
| |
| </syntaxhighlight>
| |
|
| |
| ==== Dynamic Layer Generator ====
| |
| To generate just a few layers from database, create a layer filter and a layer mixer:<syntaxhighlight lang="yaml">
| |
| gentmp:
| |
| uri: bridge://
| |
| xml:
| |
| npm: ["osm-bright-source", "data.xml"]
| |
| xmlSetDataSource:
| |
| if:
| |
| dbname: gis
| |
| host: ""
| |
| type: postgis
| |
| set:
| |
| host: localhost
| |
| user: {var: osmdb-user}
| |
| password: {var: osmdb-pswd}
| |
| xmlLayers: [admin, road]
| |
|
| |
| mixtmp:
| |
| uri: layermixer://
| |
| params:
| |
| sources: [{ref: v2}, {ref: gentmp}]
| |
| </syntaxhighlight>Once set, POST a job to copy <code>mixtmp</code> into the storage <code>v2</code> e.g.
| |
|
| |
| '''<code>src=mixtmp dst=v2 baseZoom=0 fromZoom=5 beforeZoom=6 parts=10</code>'''
| |
|
| |
| === Generating Tiles ===
| |
| Generate all tiles for zooms <code>0..7</code>, using generator <code>gen</code>, saving into <code>v4</code> everything including the solid tiles, up to 4 jobs per zoom.
| |
|
| |
| '''<code>src=gen dst=v4 parts=4 baseZoom=0 fromZoom=0 beforeZoom=8 saveSolid=1</code>'''
| |
|
| |
| Generated tiles only if they already exist in <code>v2</code> source, and save them into <code>v4</code>, on zooms <code>8..15</code>, 60 jobs per zoom.
| |
|
| |
| '''<code>src=gen dst=v4 parts=60 baseZoom=0 fromZoom=8 beforeZoom=16 sourceId=v2</code>'''
| |
|
| |
| === Bulk Copying ===
| |
| The fastest way to copy a large number of tiles from one source to another is to use a large number of parts and specify <code>saveSolid=true</code> (skips solid tile detection). E.g. to copy all z16 tiles from v3 to v4, use '''<code>src=v3 dst=v4 zoom=16 parts=60 saveSolid=true</code>'''
| |
|
| |
| == Puppetization and Automation ==
| |
| === Prerequisites ===
| |
| * passwords and postgres replication configuration is set in Ops private repo (<code>root@palladium:~/private/hieradata/role/(codfw|eqiad)/maps/server.yaml</code>)
| |
| * other configuration in <code>puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml</code>
| |
| * <code>cassandra::rack</code> is defined in <code>puppet/hieradata/hosts/maps*.yaml</code>
| |
| * the <code>role::maps::master</code> / <code>role::maps::slave</code> roles are associated to the maps nodes (site.pp)
| |
|
| |
| === Manual steps ===
| |
|
| |
| * On first start of a Cassandra node, it is necessary to clean the data directory <code>rm -rf /srv/cassandra/*</code>. See [[Cassandra#Adding_a_new_.28empty.29_node|Cassandra documentation]] for more details.
| |
| *Also remember to touch /etc/cassandra/service-enabled before starting Cassandra on each node. See https://phabricator.wikimedia.org/T214166
| |
|
| |
| * To initialize the first Cassandra node, we need to add the local node to the list of seeds by manually editing <code>/etc/cassandra/cassandra.yaml</code> and restarting cassandra:
| |
|
| |
| <syntaxhighlight lang="yaml">
| |
| seed_provider:
| |
| # Addresses of hosts that are deemed contact points.
| |
| # Cassandra nodes use this list of hosts to find each other and learn
| |
| # the topology of the ring. You must change this if you are running
| |
| # multiple nodes!
| |
| - class_name: org.apache.cassandra.locator.SimpleSeedProvider
| |
| parameters:
| |
| # seeds is actually a comma-delimited list of addresses.
| |
| # Ex: "<ip1>,<ip2>,<ip3>"
| |
| # Omit own host name / IP in multi-node clusters (see
| |
| # https://phabricator.wikimedia.org/T91617).
| |
| - seeds: 10.64.16.42,10.64.32.117,10.64.48.154 # '''add local node here to initialize the first Cassandra node'''
| |
| </syntaxhighlight>
| |
|
| |
| * change the cassandra super user password to match the one configured in private repo using cqlsh:
| |
| cqlsh '''<maps1001.eqiad.wmnet>''' -u cassandra
| |
| Password: cassandra
| |
| ALTER USER cassandra WITH PASSWORD '<password>';
| |
| * Setup of user access / rights for cassandra
| |
| cat /usr/local/bin/maps-grants.cql | cqlsh '''<maps1001.eqiad.wmnet>''' -u cassandra
| |
| * Setup replication of Cassandra <code>system_auth</code> according to [[Cassandra#Replicating_system_auth|documentation]].
| |
| *'''Very important point to note is there's a catch when setting up replicator factor for system_auth keyspace on cassandra.''' We recently discovered that increasing replication factor according to docs causes outage on cassandra. See [[phab:T214434|T214434]] and [[phab:T157354|T157354]]. Also this: [[Incident documentation/20190122-maps]].
| |
|
| |
| * Initial creation of cassandra keyspace: To prevent accidental modification of schema, Tilerator source configuration does not allow to create schema by default. The sources file used by tilerator / kartotherian is configured in <code>/etc/(kartotherian|tilerator|tileratorui)/config.yaml</code>, look for the <code>sources:</code> key. This is a reference to a sources file in the kartotherian / tilerator source directory. For example <code>/srv/deployment/tilerator/deploy/src/sources.prod2.yaml</code>.
| |
|
| |
| The easiest way to create a new keyspace is to run Tilerator with a custom sources file, which instructs tilerator to create the missing keyspace. For example, create a temporary file, e.g. <code>/home/yurik/my-source-file</code> with the following configuration (change <code>v4</code> with the keyspace declared in the sources configuration file):
| |
| <syntaxhighlight lang="yaml">
| |
| v4:
| |
| uri: cassandra://
| |
| params:
| |
| keyspace: v4
| |
| cp: {var: cassandra-servers}
| |
| username: {var: cassandra-user}
| |
| password: {var: cassandra-pswd}
| |
| # These parameters are only used if keyspace needs to be created:
| |
| repfactor: 4
| |
| durablewrite: 0
| |
| createIfMissing: true
| |
| copyInfoFrom: {ref: gen}
| |
| </syntaxhighlight>
| |
|
| |
| And run this bash script:
| |
| <syntaxhighlight lang="bash">
| |
| node /srv/deployment/tilerator/deploy/node_modules/tilerator/scripts/tileshell.js \
| |
| # Use TileratorUI configuration (including password variables)
| |
| --config /etc/tileratorui/config.yaml \
| |
| # Use this sources file instead of the one in the config file
| |
| --source /home/yurik/my-source-file
| |
| </syntaxhighlight>
| |
|
| |
| Tileshell will not exit, so ^C it after it reports "done".
| |
|
| |
| * On existing server, record all existing tiles as a list of tile indexes (path and generatorId need to be adapted)
| |
| <syntaxhighlight lang="bash">
| |
| node /srv/deployment/tilerator/deploy/node_modules/tilerator/scripts/tileshell.js \
| |
| --config /etc/tileratorui/config.yaml \
| |
| # List all tiles in the "v5" source
| |
| -j.generatorId v5 \
| |
| # Which zoom to enumerate
| |
| -j.zoom 14 \
| |
| # File to write indexes to
| |
| --dumptiles /home/yurik/all-tiles-14.txt \
| |
| # Instead of dumping indexes in "zoom/x/y" format, write one number indexes (0..4^zoom-1)
| |
| --dumprawidx \
| |
| # If dumptiles file already exists, override it
| |
| --dumpoverride
| |
| </syntaxhighlight>
| |
| * Instead of generating the entire zoom level, you may want to generate just the tiles in a list (all parameters might need to be adapted)
| |
| <syntaxhighlight lang="bash">
| |
| node /srv/deployment/tilerator/deploy/node_modules/tilerator/scripts/tileshell.js \
| |
| --config /etc/tileratorui/config.yaml \
| |
| # List of tile indexes, unique and sorted, one per line.
| |
| # Indexes can be 0..4^zoom-1
| |
| -j.filepath /home/yurik/all-tiles-14.txt \
| |
| # All tile indexes in the file belong to zoom 14
| |
| # Without this parameter, the file must contain zoom/x/y triplets
| |
| -j.fileZoomOverride 14 \
| |
| # generate zoom levels 10 <= zoom < 16
| |
| -j.fromZoom 10 -j.beforeZoom 16 \
| |
| # Copy tiles from "gen" source to "v4" source
| |
| -j.generatorId gen -j.storageId v4 \
| |
| # If tile already exists in "v4", but "gen" produces an empty tile, delete it in v4
| |
| -j.deleteEmpty
| |
| </syntaxhighlight>
| |
|
| |
| == Deploying maps services ==
| |
|
| |
| === Git repositories galore ===
| |
| Kartotherian is developed [https://github.com/kartotherian on GitHub], however for deployment purposes we have copies of both services' main repos in Gerrit, at <code>[https://gerrit.wikimedia.org/r/#/admin/projects/maps/kartotherian maps/kartotherian]</code> and <code>[https://gerrit.wikimedia.org/r/#/admin/projects/maps/tilerator maps/tilerator]</code>. All development should happen on GitHub while Gerrit can have WMF-specific hacks (that should be avoided, however).
| |
|
| |
| ==== Keeping repositories in sync ====
| |
| Assuming you have a checkout from Gerrit, create a remote called <code>github</code>:
| |
|
| |
| kartotherian$ git remote add github https://github.com/kartotherian/kartotherian.git
| |
|
| |
| Then when you need to sync the repositories, pull from GitHub and push to Gerrit:
| |
|
| |
| git pull github master
| |
| git push
| |
|
| |
| === Building for deployment ===
| |
| Kartotherian and Tilerator are deployed according to the [[mw:ServiceTemplateNode/Deployment|standard process for deploying]] Wikimedia Node.js services, with the important difference that deployments are built from purpose-specific "package" repos rather than directly from service code repos. This is to facilitate the bundling of additional maps-specific dependencies.
| |
|
| |
| Kartotherian: https://gerrit.wikimedia.org/r/#/admin/projects/maps/kartotherian/package<br>
| |
| Tilerator: https://gerrit.wikimedia.org/r/#/admin/projects/maps/tilerator/package
| |
|
| |
| === Deploying services ===
| |
| Refer to [[Services/Deployment]] for general instructions.
| |
|
| |
| In most cases, Kartotherian and Tilerator should be deployed together, to ensure that all dependencies (in particular, styles) are in sync.
| |
|
| |
|
| == Subpages == | | == Subpages == |
| {{Special:PrefixIndex/{{PAGENAME}}/|hideredirects=1|stripprefix=1}} | | {{Special:PrefixIndex/{{PAGENAME}}/|hideredirects=1|stripprefix=1}} |
