You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Maps/Kartotherian: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>MSantos
(Created page with "== [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] == Kartotherian serves map tiles by getting vector data from Cassandra, applying the [http...")
 
imported>Quiddity
(lang="...")
 
(One intermediate revision by one other user not shown)
Line 1: Line 1:
== [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] ==
== [https://github.com/kartotherian/kartotherian/blob/master/README.md Kartotherian] - Maps Tile service for Wikipedia ==
 
Kartotherian serves map tiles by getting vector data from Cassandra, applying the [https://github.com/kartotherian/osm-bright.tm2/blob/master/README.md style] to it, and returning raster images. It is also capable of serving a "static image" - a map with a given width/height/scaling/zoom, and can server vector tiles directly for on-the-client rendering (WebGL maps).
Kartotherian serves map tiles by getting vector data from Cassandra, applying the [https://github.com/kartotherian/osm-bright.tm2/blob/master/README.md style] to it, and returning raster images. It is also capable of serving a "static image" - a map with a given width/height/scaling/zoom, and can server vector tiles directly for on-the-client rendering (WebGL maps).
* [Https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/kartotherian/templates/config.yaml.erb Service configuration template] (extends [[phab:diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb|base template]])
* [Https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/kartotherian/templates/config.yaml.erb Service configuration template] (extends [[phab:diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb|base template]])
* [https://phabricator.wikimedia.org/diffusion/GMKT/browse/master/sources.prod.yaml Sources configuration]
* [https://phabricator.wikimedia.org/diffusion/GMKT/browse/master/sources.prod.yaml Sources configuration]
To see the tiles without Varnish cache, connect to Kartotherian using an ssh tunnel, e.g. <code>ssh -L 6533:localhost:6533 maps-test2001.codfw.wmnet</code> and browse to http://localhost:6533
To see the tiles without Varnish cache, connect to Kartotherian using an ssh tunnel, e.g. <code>ssh -L 6533:localhost:6533 maps-test2001.codfw.wmnet</code> and browse to http://localhost:6533
Maps nodejs server for vector-based tiles and snapshots, designed for Wikipedia and other sites. It ties together a number of [https://github.com/mapbox MapBox] components for vector and raster rendering based on [https://github.com/mapnik/mapnik Mapnik 3] , and uses [https://github.com/wikimedia/service-runner service runner]  for scalability, performance monitoring and stability.
===  Serving tiles ===
Kartotherian can serve vector and raster tiles in multiple formats and optional scaling:
    http://.../{source}/{zoom}/{x}/{y}[@{scale}x].{format}
* The sources are configured with the
[https://github.com/kartotherian/core source config file] . Sources configuration supports different methods of tile storage, such as Cassandra or files, generation from postgress db, overzoom to extract the tile from lower zooms if missing, layer extraction, mixing multiple sources together, etc.
* Optional scalling can render larger images for high resolution screens (only those enabled in the source, e.g. <CODE>[1.5, 2]</CODE>)
* Supported formats include PNG ang JPEG, SVG, PBF vectors, and JSON (with<code>nogeo</code> and <code>summary</code>debug options)
===  Static map images ===
Kartotherian supports static image generation. Users may request a PNG or a JPEG snapshot image of any size, scaling, and zoom level:
    http://.../img/{source},{zoom},{lat},{lon},{width}x{height}[@{scale}x].{format}
    # image centered at 42,-3.14, at zoom level 4, size 800x600
    http://.../img/osm-intl,4,42,-3.14,800x600.png
    # the same but for higher DPI device with 1.5 scaling
    http://.../img/osm-intl,4,42,-3.14,800x600@1.5x.png
===  Info data ===
Kartotherian can be used as a source of the PBF data for Mapbox studio. See info about style editing in  [https://github.com/kartotherian/osm-bright.tm2/blob/master/README.md osm-bright-source] . The info data is available at<code>http://.../{style}/pbfinfo.json</code> for pbf source, and <code>http://.../{style}/info.json</code>for the styled image source.
===  Markers ===
Kartotherian can generate marker images by wrapping any of the [https://www.mapbox.com/maki/ maki icons]  with a pushpin image, in any color. The URL schema is matched to the one used by the [https://github.com/mapbox/mapbox.js mapbox.js] .
    http://.../v4/marker/pin-l-cafe+de00ff@2x.png
    http://.../v4/marker/ {base} - {size:s|m|l} [-{letter-or-digit-or-icon-name}] + {color} [@2x] .png
At this point, only "pin" is supported for the base. The color is a 3 digit or 6 digit hex number. Optional scaling can only be 2x. Beyond the pre-defined maki icons, you may give a number (0-99), a single letter (a-z), or nothing.
==  Very quick start ==
<syntaxhighlight lang="bash">
git clone https://github.com/kartotherian/kartotherian.git  # Clone the repository
cd kartotherian
</syntaxhighlight>
<syntaxhighlight lang="bash">
npm install
node server.js -c config.external.yaml
</syntaxhighlight>
Browse to http://localhost:6533/
The set up inside [https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/kartotherian/+/refs/heads/master/packages/kartotherian/sources.external.yaml <code>sources.external.yaml</code>]  does not use any storage or caching, so it will not be suitable for production. You will need to to set up your own local database as described in [https://github.com/kartotherian/osm-bright.tm2source osm-bright.tm2source] , which is installed in <code>node_modules/osm-bright-source</code>, and configure additional source chains and setup a proper storage to make this into a production system.
==  Configuration ==
Inside the<code>conf</code>key:
*<code>sources</code>- (required) Either a set of subkeys, a filename, or a list of file names.  See [https://github.com/kartotherian/core core]  on how to configure the sources.
*<code>variables</code>(optional) - specify a set of variables (string key-value pairs) to be used inside sources, or it could be a filename or a list of filenames/objects.
*<code>defaultHeaders</code>(optional, object) - a set of extra headers that will be sent to the user unless the source provides its own. (public requests only)
*<code>headers</code>(optional, object) - a set of extra headers that will be sent to the user instead of the headers returned by the source. (public requests only)
For the rest of the configuration parameters, see [https://github.com/wikimedia/service-runner service runner]  config info.
==  Components ==
Kartotherian platform consists of a number of elements, some of which conform to the general specifications established
by [https://github.com/mapbox MapBox] , and therefor can reuse components that confirm to the same specification.
Also, see [https://github.com/kartotherian/tilerator Tilerator] , an optional stand-alone service to pre-generate tiles.
Tilerator is separate from Kartotherian, but it reuses most of the same components.
===  Components by Wikimedia Foundation ===
* [https://github.com/kartotherian/core kartotherian-core]  - Loads and configures tile sources, and provides some common utility functions
* [https://github.com/kartotherian/server kartotherian-server]  - Handles user requests for tiles and source info, as well as registers additional data type handlers like maki markers and image snapshots.
* [https://github.com/kartotherian/maki kartotherian-maki]  - Request handler for maki markers - generates PNG marker images that can be used from geojson.
* [https://github.com/kartotherian/snapshot kartotherian-snapshot]  - Request handler for static images by combining multiple tiles into one snapshot image of a requested size.
====  Tile sources ====
* [https://github.com/kartotherian/autogen kartotherian-autogen]  - Tile source that checks "storage" source for a tile, and if not found, gets it from the "generator" source and saves it into the "storage"
* [https://github.com/kartotherian/cassandra kartotherian-cassandra]  - Tile source that stores tiles in the Cassandra database
* [https://github.com/kartotherian/demultiplexer kartotherian-demultiplexer]  - Tile source that combines multiple sources by zoom level
* [https://github.com/kartotherian/layermixer kartotherian-layermixer]  - Tile source capable of mixing different vector layers from multiple tile sources
* [https://github.com/kartotherian/overzoom kartotherian-overzoom]  - Tile source that will zoom out if the requested tile does not exist, and extracts the needed portion from the lower-zoom tile it finds.
* [https://github.com/kartotherian/postgres kartotherian-postgres]  - Tile source that stores tiles in the Postgres database
* [https://github.com/kartotherian/substantial kartotherian-substantial]  - Tile source that filters out tiles that are not significant - e.g. nothing but water or land.
====  Data and Styling ====
* [https://github.com/kartotherian/osm-bright.tm2source osm-bright-source]  - SQL queries used by the<code>tilelive-bridge</code>to generate a vector tile from Postgres Database
* [https://github.com/kartotherian/osm-bright.tm2 osm-bright-style]  - Style used by the<code>tilelive-vector</code>to convert vector tiles into images.
* [https://github.com/kartotherian/osm-bright.fonts osm-bright-fonts]  - Fonts used by the <code>osm-bright-style</code>.
===  Components by MapBox ===
* [https://github.com/mapbox/tilelive tilelive]  - ties together various tile sources, both vector and raster
* [https://github.com/mapbox/tilelive-bridge tilelive-bridge]  - generates vector tiles from SQL
* [https://github.com/mapbox/tilelive-vector tilelive-vector]  - converts vector tiles to raster tiles
* [https://github.com/mapbox/abaculus abaculus]  - generates raster images of any location and size from a tile source
===  Other Relevant Components ===
* [https://github.com/mapnik/node-mapnik mapnik]  - Tile rendering library for node
* [https://github.com/Leaflet/Leaflet leaflet]  - JavaScript library for mobile-friendly interactive maps
==  In depth step-by-step: ==
This documentation assumes that you are going to use [https://github.com/kartotherian/osm-bright.tm2 https://github.com/kartotherian/osm-bright.tm2source osm-bright.tm2]  and [osm-bright.tm2source]  for a map style.
===  Install node.js and npm ===
Node.js v6 or v8 required; v10+ currently not supported.
For Windows and Mac, downloadable installers can be found at https://nodejs.org/download/release/latest-v6.x/ or https://nodejs.org/download/release/latest-v8.x/.
For Linux, installation via the instructions at https://nodejs.org/en/download/package-manager/ is recommended.
===  Install other software dependencies ===
On Ubuntu these can be installed with
<syntaxhighlight lang="bash">
sudo apt-get install git unzip curl build-essential sqlite3 pkg-config libcairo2-dev libjpeg-dev libgif-dev libmapnik-dev
</syntaxhighlight>
===  Clone the repo and install npm dependencies ===
<syntaxhighlight lang="bash">
git clone https://github.com/kartotherian/kartotherian.git
cd kartotherian
npm install
</syntaxhighlight>
===  Source ===
Set up osm-bright.tm2source as described in [https://github.com/kartotherian/osm-bright.tm2source#install its documentation.] .
osm-bright.tm2source is installed in <code>node_modules/osm-bright-source</code>
===  Edit Kartotherian configuration - config.yaml ===
<syntaxhighlight lang="yaml">
# 0 - one instance, 1+ - multi-instance with autorestart, ncpu - multi-instance, one per CPU
num_workers: 0
# Host port
port: 6533
# Comment out this line to listen to the web
# interface: localhost
# Place all variables (e.g. passwords) here - either as a filename, or as sub-items.
variables:
# Place all sources you want to serve here - either as a filename, or as sub-items.
# See sources.prod.yaml for examples
sources: sources.yaml
</syntaxhighlight>
===  Configure Kartotherian ===
Use one of the config files, or update them, and make a link config.yaml to it.
===  Add Varnish caching layer (optional) ===
Might require caching headers added to the source/config.
<syntaxhighlight lang="bash">
# From https://www.varnish-cache.org/installation/debian
sudo -Hi
apt-get install apt-transport-https
curl https://repo.varnish-cache.org/GPG-key.txt | apt-key add -
echo "deb https://repo.varnish-cache.org/debian/ jessie varnish-4.0" >> /etc/apt/sources.list.d/varnish-cache.list
apt-get update
apt-get install varnish
vi /etc/varnish/default.vcl
</syntaxhighlight>
Change default backend to:
<syntaxhighlight lang="text">
backend default {
    .host = "localhost";
    .port = "6533";
}
</syntaxhighlight>
Add this to vcl_deliver (to track hits/misses):
<syntaxhighlight lang="text">
if (obj.hits > 0) {
    set resp.http.X-Cache = "HIT";
} else {
    set resp.http.X-Cache = "MISS";
}
</syntaxhighlight>
Edit /etc/systemd/system/varnish.service - set proper listening port (80) and cache size:
<syntaxhighlight lang="text">
ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,4g
</syntaxhighlight>
In bash:
<syntaxhighlight lang="bash">
systemctl daemon-reload  # because we changed the .service file
systemctl restart varnish.service
systemctl status varnish.service  # check the service started with the right params
varnishstat  # monitor varnish performance
</syntaxhighlight>
===  Run Karthotherian: ===
<syntaxhighlight lang="bash">
npm start
</syntaxhighlight>
In browser, navigate to <code>http://localhost:6533/</code>.
===  Troubleshooting ===
In a lot of cases when there is an issue with node it helps to recreate the
<code>node_modules</code> directory:
<syntaxhighlight lang="bash">
rm -r node_modules
npm install
</syntaxhighlight>

Latest revision as of 02:12, 8 August 2021

Kartotherian - Maps Tile service for Wikipedia

Kartotherian serves map tiles by getting vector data from Cassandra, applying the style to it, and returning raster images. It is also capable of serving a "static image" - a map with a given width/height/scaling/zoom, and can server vector tiles directly for on-the-client rendering (WebGL maps).

To see the tiles without Varnish cache, connect to Kartotherian using an ssh tunnel, e.g. ssh -L 6533:localhost:6533 maps-test2001.codfw.wmnet and browse to http://localhost:6533 Maps nodejs server for vector-based tiles and snapshots, designed for Wikipedia and other sites. It ties together a number of MapBox components for vector and raster rendering based on Mapnik 3 , and uses service runner for scalability, performance monitoring and stability.

Serving tiles

Kartotherian can serve vector and raster tiles in multiple formats and optional scaling:

   http://.../{source}/{zoom}/{x}/{y}[@{scale}x].{format}
  • The sources are configured with the

source config file . Sources configuration supports different methods of tile storage, such as Cassandra or files, generation from postgress db, overzoom to extract the tile from lower zooms if missing, layer extraction, mixing multiple sources together, etc.

  • Optional scalling can render larger images for high resolution screens (only those enabled in the source, e.g. [1.5, 2])
  • Supported formats include PNG ang JPEG, SVG, PBF vectors, and JSON (withnogeo and summarydebug options)

Static map images

Kartotherian supports static image generation. Users may request a PNG or a JPEG snapshot image of any size, scaling, and zoom level:

   http://.../img/{source},{zoom},{lat},{lon},{width}x{height}[@{scale}x].{format}
   # image centered at 42,-3.14, at zoom level 4, size 800x600
   http://.../img/osm-intl,4,42,-3.14,800x600.png
   # the same but for higher DPI device with 1.5 scaling
   http://.../img/osm-intl,4,42,-3.14,800x600@1.5x.png

Info data

Kartotherian can be used as a source of the PBF data for Mapbox studio. See info about style editing in osm-bright-source . The info data is available athttp://.../{style}/pbfinfo.json for pbf source, and http://.../{style}/info.jsonfor the styled image source.

Markers

Kartotherian can generate marker images by wrapping any of the maki icons with a pushpin image, in any color. The URL schema is matched to the one used by the mapbox.js .

   http://.../v4/marker/pin-l-cafe+de00ff@2x.png
   http://.../v4/marker/ {base} - {size:s|m|l} [-{letter-or-digit-or-icon-name}] + {color} [@2x] .png

At this point, only "pin" is supported for the base. The color is a 3 digit or 6 digit hex number. Optional scaling can only be 2x. Beyond the pre-defined maki icons, you may give a number (0-99), a single letter (a-z), or nothing.

Very quick start

git clone https://github.com/kartotherian/kartotherian.git  # Clone the repository
cd kartotherian
npm install
node server.js -c config.external.yaml

Browse to http://localhost:6533/ The set up inside sources.external.yaml does not use any storage or caching, so it will not be suitable for production. You will need to to set up your own local database as described in osm-bright.tm2source , which is installed in node_modules/osm-bright-source, and configure additional source chains and setup a proper storage to make this into a production system.


Configuration

Inside theconfkey:

  • sources- (required) Either a set of subkeys, a filename, or a list of file names. See core on how to configure the sources.
  • variables(optional) - specify a set of variables (string key-value pairs) to be used inside sources, or it could be a filename or a list of filenames/objects.
  • defaultHeaders(optional, object) - a set of extra headers that will be sent to the user unless the source provides its own. (public requests only)
  • headers(optional, object) - a set of extra headers that will be sent to the user instead of the headers returned by the source. (public requests only)

For the rest of the configuration parameters, see service runner config info.

Components

Kartotherian platform consists of a number of elements, some of which conform to the general specifications established by MapBox , and therefor can reuse components that confirm to the same specification. Also, see Tilerator , an optional stand-alone service to pre-generate tiles. Tilerator is separate from Kartotherian, but it reuses most of the same components.

Components by Wikimedia Foundation

  • kartotherian-core - Loads and configures tile sources, and provides some common utility functions
  • kartotherian-server - Handles user requests for tiles and source info, as well as registers additional data type handlers like maki markers and image snapshots.
  • kartotherian-maki - Request handler for maki markers - generates PNG marker images that can be used from geojson.
  • kartotherian-snapshot - Request handler for static images by combining multiple tiles into one snapshot image of a requested size.

Tile sources

  • kartotherian-autogen - Tile source that checks "storage" source for a tile, and if not found, gets it from the "generator" source and saves it into the "storage"
  • kartotherian-cassandra - Tile source that stores tiles in the Cassandra database
  • kartotherian-demultiplexer - Tile source that combines multiple sources by zoom level
  • kartotherian-layermixer - Tile source capable of mixing different vector layers from multiple tile sources
  • kartotherian-overzoom - Tile source that will zoom out if the requested tile does not exist, and extracts the needed portion from the lower-zoom tile it finds.
  • kartotherian-postgres - Tile source that stores tiles in the Postgres database
  • kartotherian-substantial - Tile source that filters out tiles that are not significant - e.g. nothing but water or land.

Data and Styling

  • osm-bright-source - SQL queries used by thetilelive-bridgeto generate a vector tile from Postgres Database
  • osm-bright-style - Style used by thetilelive-vectorto convert vector tiles into images.
  • osm-bright-fonts - Fonts used by the osm-bright-style.


Components by MapBox

  • tilelive - ties together various tile sources, both vector and raster
  • tilelive-bridge - generates vector tiles from SQL
  • tilelive-vector - converts vector tiles to raster tiles
  • abaculus - generates raster images of any location and size from a tile source

Other Relevant Components

  • mapnik - Tile rendering library for node
  • leaflet - JavaScript library for mobile-friendly interactive maps

In depth step-by-step:

This documentation assumes that you are going to use https://github.com/kartotherian/osm-bright.tm2source osm-bright.tm2 and [osm-bright.tm2source] for a map style.

Install node.js and npm

Node.js v6 or v8 required; v10+ currently not supported.

For Windows and Mac, downloadable installers can be found at https://nodejs.org/download/release/latest-v6.x/ or https://nodejs.org/download/release/latest-v8.x/.

For Linux, installation via the instructions at https://nodejs.org/en/download/package-manager/ is recommended.

Install other software dependencies

On Ubuntu these can be installed with

sudo apt-get install git unzip curl build-essential sqlite3 pkg-config libcairo2-dev libjpeg-dev libgif-dev libmapnik-dev

Clone the repo and install npm dependencies

git clone https://github.com/kartotherian/kartotherian.git
cd kartotherian
npm install

Source

Set up osm-bright.tm2source as described in its documentation. .

osm-bright.tm2source is installed in node_modules/osm-bright-source

Edit Kartotherian configuration - config.yaml

# 0 - one instance, 1+ - multi-instance with autorestart, ncpu - multi-instance, one per CPU
num_workers: 0

# Host port
port: 6533

# Comment out this line to listen to the web
# interface: localhost

# Place all variables (e.g. passwords) here - either as a filename, or as sub-items.
variables:

# Place all sources you want to serve here - either as a filename, or as sub-items.
# See sources.prod.yaml for examples
sources: sources.yaml

Configure Kartotherian

Use one of the config files, or update them, and make a link config.yaml to it.

Add Varnish caching layer (optional)

Might require caching headers added to the source/config.

# From https://www.varnish-cache.org/installation/debian
sudo -Hi
apt-get install apt-transport-https
curl https://repo.varnish-cache.org/GPG-key.txt | apt-key add -
echo "deb https://repo.varnish-cache.org/debian/ jessie varnish-4.0" >> /etc/apt/sources.list.d/varnish-cache.list
apt-get update
apt-get install varnish

vi /etc/varnish/default.vcl

Change default backend to:

backend default {
    .host = "localhost";
    .port = "6533";
}

Add this to vcl_deliver (to track hits/misses):

if (obj.hits > 0) {
    set resp.http.X-Cache = "HIT";
} else {
    set resp.http.X-Cache = "MISS";
}

Edit /etc/systemd/system/varnish.service - set proper listening port (80) and cache size:

ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,4g

In bash:

systemctl daemon-reload  # because we changed the .service file
systemctl restart varnish.service
systemctl status varnish.service  # check the service started with the right params
varnishstat  # monitor varnish performance

Run Karthotherian:

npm start

In browser, navigate to http://localhost:6533/.

Troubleshooting

In a lot of cases when there is an issue with node it helps to recreate the node_modules directory:

rm -r node_modules
npm install