You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "Maps/v2/Common tasks"

From Wikitech-static
< Maps‎ | v2
Jump to navigation Jump to search
imported>Jgiannelos
 
imported>Quiddity
(lang=bash)
 
(2 intermediate revisions by 2 users not shown)
Line 2: Line 2:


=== Access map nodes ===
=== Access map nodes ===
<code>ssh maps1009.eqiad.wmnet</code>


=== Deploy new code ===
=== Deploy new code ===


==== Kartotherian ====
==== Kartotherian ====
Currently, kartotherian main code is available in the imposm branch.
In order to deploy kartotherian you need to package the deploy repo locally and submit to gerrit, kartotherian uses service-runner to run a Docker container for packaging, the following script assumes you have both the package repo and the deploy repo at the same directory level:
<syntaxhighlight lang=bash>
cd ../package
git status
git reset --hard origin/imposm
git checkout .
git clean -fd
git checkout imposm
git pull origin imposm
git status
git --no-pager log --decorate -n 1
rm -rf node_modules/
npm i service-runner
cd ../deploy
git status
git reset --hard origin/imposm
git checkout .
git clean -fd
git checkout imposm
git pull
git --no-pager log --decorate -n 1
git submodule update --init
git status
cd ../package
./server.js build --deploy-repo --force
</syntaxhighlight>


===== Tegola =====
===== Tegola =====
*Build a new image using CI
** Each merge builds an image:
*** https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/tegola/+/refs/heads/wmf/v0.14.x
*** https://docker-registry.wikimedia.org/wikimedia/operations-software-tegola/tags/
* Bump version on [[gerrit:plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/services/tegola-vector-tiles/|deployment-charts]]
* [[Kubernetes/Deployments|Deploy]] new version


=== Configuration changes ===
=== Configuration changes ===


==== Tegola ====
==== Tegola ====
The tegola config is maintained using helm in the deployment charts repo and populated by the helm values per env:
* Config
** https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/charts/tegola-vector-tiles/templates/_config.tpl
* Values
** Base values
*** https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/charts/tegola-vector-tiles/values.yaml
** Per env overrides
*** https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/services/tegola-vector-tiles/


==== Kartotherian ====
==== Kartotherian ====
Kartotherian is using scap for deployment. The configuration is maintained here:
* https://gerrit.wikimedia.org/r/plugins/gitiles/maps/kartotherian/deploy/+/refs/heads/master/scap/templates/config.yaml.j2
* https://gerrit.wikimedia.org/r/plugins/gitiles/maps/kartotherian/deploy/+/refs/heads/master/scap/vars.yaml


==== Postgres ====
==== Postgres ====
Postgres config is maintained using puppet. Here are the relevant sections:
* https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/postgresql/


==== Imposm ====
==== Imposm ====
OSM import process is configured using puppet. Here are the relevant sections:
* https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/osm/


==== EventGate ====
==== EventGate ====
* Schema definitions
** https://gerrit.wikimedia.org/r/plugins/gitiles/schemas/event/primary/+/refs/heads/master/jsonschema/maps/tile_change
** https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/wmf-config/InitialiseSettings.php
*** Go to section "Stream config"
=== Enable/disable import process ===
In order to disable the import process you need to switch the boolean value to false and run puppet:
* <code>profile::maps::osm_master::disable_replication_cron: false</code>


=== Run a planet import ===
=== Run a planet import ===
When running a planet import, make sure that the following checklist is followed:
* disable disable_replication_service and disable_tile_generation_cron in hieradata
* execute initial-imposm-import script using the following command
<syntaxhighlight lang=bash>
# change -d accordingly when performing the import with the most updated values
imposm-initial-import -d 210906 -x webproxy.eqiad.wmnet:8080
</syntaxhighlight>
* monitor the full planet import
** Is there any disk space issue?
** Did the script finished properly?
** Are the logs sufficient?
* monitor and check the APIs, if needed rollback with the following command
<syntaxhighlight lang=bash>
sudo -u osmupdater imposm-rollback-import
</syntaxhighlight>
* re-enable the replication service and check for its health
* re-enable the tile-generation cron and check for its health
* remove imposm backup to free disk space
<syntaxhighlight lang=bash>
sudo -u osmupdater imposm-removebackup-import
</syntaxhighlight>


=== Pooling/depooling nodes ===
=== Pooling/depooling nodes ===
TODO


=== Get a DB shell ===
=== Get a DB shell ===
<code>> ssh maps1005.eqiad.wmnet</code>
<code>> sudo su </code>
<code>> su - postgres </code>
<code>> psql </code>
<code>postgres=# \c gis</code>


=== Run an example query ===
=== Run an example query ===
Select all landuse geometries included in the bounding box:
* [[:en:Spatial_reference_system|SRID]] 3857 (the one we use in our layers config)
* <code>Box: left=2632692 bottom=4572168 right=2648887 top=4581940</code>
This bounding box is a rectangle that includes the center of Athens, Greece.
<code>gis=# SELECT osm_id, ST_AsMVTGeom(geometry, ST_MakeEnvelope(2632692,4572168,2648887,4581940, 3857)) AS geom FROM layer_landuse(ST_MakeEnvelope(2632692,4572168,2648887,4581940, 3857), 9);</code>


=== Reindex DB ===
=== Reindex DB ===
<blockquote>WARNING this operation might take even ~10 hours to be completed</blockquote>From a pgsql shell, connect to the `gis` DB and run:
<code>gis #= REINDEX DATABASE gis;</code>


=== Example HTTP requests ===
=== Example HTTP requests ===


==== Kartotherian ====
==== Kartotherian ====
Get the raster tile <code>x=0 y=0 z=0</code> for source <code>osm-intl</code>
<code>curl https://maps.wikimedia.org/osm-intl/0/0/0.png</code>


==== Tegola ====
==== Tegola ====
Tegola currently is only used internally. In order to reach out for the current example we will ssh to the deployment node (eq. in eqiad)
<code>> ssh deployment.eqiad.wmnet </code>
To get the endpoint information for different maps and layers:
<code>> curl https://tegola-vector-tiles.svc.eqiad.wmnet:4105/capabilities | jq .</code>


To get the vector tile <code>x=0 y=0 z=0</code> for map <code>osm</code>
<code>> curl https://tegola-vector-tiles.svc.eqiad.wmnet:4105/maps/osm/{z}/{x}/{y}.pbf</code>
==== EventGate ====
==== EventGate ====
Post an event:
<syntaxhighlight lang=bash>
curl -X POST -d '
{
    "\$schema": "/maps/tile_change/1.0.0",
    "meta": {
   "dt": "$(date --iso-8601=s)"
    "stream": "maps.tile_expiration"
  },
"tile": "1/0/2",
"state": "expired"
}' https://eventgate-main.svc.eqiad.wmnet:4492/v1/events/
</syntaxhighlight>


=== Example tegola commands ===
=== Example tegola commands ===
* Run server
** <code>tegola -c </path/to/config> serve</code>
* Cache a tile
** <code>tegola -c /path/to/config cache seed tile-name 0/0/0</code>
* Purge a tile from cache
** <code>tegola -c /path/to/config cache purge tile-name 0/0/0</code>
[[Category:Maps V2]]
[[Category:Maps V2]]

Latest revision as of 23:06, 1 October 2021

Common maps tasks

Access map nodes

ssh maps1009.eqiad.wmnet

Deploy new code

Kartotherian

Currently, kartotherian main code is available in the imposm branch.

In order to deploy kartotherian you need to package the deploy repo locally and submit to gerrit, kartotherian uses service-runner to run a Docker container for packaging, the following script assumes you have both the package repo and the deploy repo at the same directory level:

cd ../package
git status
git reset --hard origin/imposm
git checkout .
git clean -fd
git checkout imposm
git pull origin imposm
git status
git --no-pager log --decorate -n 1
rm -rf node_modules/
npm i service-runner

cd ../deploy
git status
git reset --hard origin/imposm
git checkout .
git clean -fd
git checkout imposm
git pull
git --no-pager log --decorate -n 1
git submodule update --init
git status

cd ../package
./server.js build --deploy-repo --force
Tegola

Configuration changes

Tegola

The tegola config is maintained using helm in the deployment charts repo and populated by the helm values per env:

Kartotherian

Kartotherian is using scap for deployment. The configuration is maintained here:

Postgres

Postgres config is maintained using puppet. Here are the relevant sections:

Imposm

OSM import process is configured using puppet. Here are the relevant sections:

EventGate

Enable/disable import process

In order to disable the import process you need to switch the boolean value to false and run puppet:

  • profile::maps::osm_master::disable_replication_cron: false

Run a planet import

When running a planet import, make sure that the following checklist is followed:

  • disable disable_replication_service and disable_tile_generation_cron in hieradata
  • execute initial-imposm-import script using the following command
# change -d accordingly when performing the import with the most updated values
imposm-initial-import -d 210906 -x webproxy.eqiad.wmnet:8080
  • monitor the full planet import
    • Is there any disk space issue?
    • Did the script finished properly?
    • Are the logs sufficient?
  • monitor and check the APIs, if needed rollback with the following command
sudo -u osmupdater imposm-rollback-import
  • re-enable the replication service and check for its health
  • re-enable the tile-generation cron and check for its health
  • remove imposm backup to free disk space
sudo -u osmupdater imposm-removebackup-import

Pooling/depooling nodes

TODO

Get a DB shell

> ssh maps1005.eqiad.wmnet

> sudo su

> su - postgres

> psql

postgres=# \c gis

Run an example query

Select all landuse geometries included in the bounding box:

  • SRID 3857 (the one we use in our layers config)
  • Box: left=2632692 bottom=4572168 right=2648887 top=4581940

This bounding box is a rectangle that includes the center of Athens, Greece.

gis=# SELECT osm_id, ST_AsMVTGeom(geometry, ST_MakeEnvelope(2632692,4572168,2648887,4581940, 3857)) AS geom FROM layer_landuse(ST_MakeEnvelope(2632692,4572168,2648887,4581940, 3857), 9);

Reindex DB

WARNING this operation might take even ~10 hours to be completed

From a pgsql shell, connect to the `gis` DB and run:

gis #= REINDEX DATABASE gis;

Example HTTP requests

Kartotherian

Get the raster tile x=0 y=0 z=0 for source osm-intl curl https://maps.wikimedia.org/osm-intl/0/0/0.png

Tegola

Tegola currently is only used internally. In order to reach out for the current example we will ssh to the deployment node (eq. in eqiad)

> ssh deployment.eqiad.wmnet

To get the endpoint information for different maps and layers:

> curl https://tegola-vector-tiles.svc.eqiad.wmnet:4105/capabilities | jq .

To get the vector tile x=0 y=0 z=0 for map osm

> curl https://tegola-vector-tiles.svc.eqiad.wmnet:4105/maps/osm/{z}/{x}/{y}.pbf

EventGate

Post an event:

curl -X POST -d '
{
    "\$schema": "/maps/tile_change/1.0.0",
    "meta": {
    "dt": "$(date --iso-8601=s)"
    "stream": "maps.tile_expiration"
  },
"tile": "1/0/2",
"state": "expired"
}' https://eventgate-main.svc.eqiad.wmnet:4492/v1/events/

Example tegola commands

  • Run server
    • tegola -c </path/to/config> serve
  • Cache a tile
    • tegola -c /path/to/config cache seed tile-name 0/0/0
  • Purge a tile from cache
    • tegola -c /path/to/config cache purge tile-name 0/0/0