Kartotherian
Map Tile Pre-Generation Service forTilerator (Russian: Тилератор, tee-LEH-ruh-tor)
This code is cross-hosted at gerrit
Generating tiles from the SQL queries sometimes requires a considerable time, often too long for the web request. Tilerator is a multi-processor, cluster-enabled tile generator, that allows both pre-generation and dirty tile re-generation.
Scheduled generation job waits in the queue (kue) until it is picked up by one of the workers. The worker will update job progress, as well as store intermediate data to allow restarts/crash recovery.
Tilerator is an unprotected Admin tool, and should NOT be exposed to the web. By default, Tilerator only accepts connections from the localhost. It is recommended that it is left this way, and used via a port forwarding ssh tunnel with ssh -L 6534:localhost:6534 my.maps.server
Tilerator also has a tileshell
script to perform some of the admin tasks without using the http protocol.
Configuration
Inside the conf
key:
sources
- (required) Either a set of subkeys, a filename, or a list of file names. See core on how to configure the sources.variables
- (optional) specify a set of variables (string key-value pairs) to be used inside sources, or it could be a filename or a list of filenames/objects.uiOnly
- (optional, boolean) runs Tilerator in UI mode - does not generate tiles, but still allows access to the web-based queue management tools.daemonOnly
- (optional, boolean) runs Tilerator in daemon mode - generates tiles, but does not allows access to the web-based queue management tools. For the rest of the configuration parameters, see service runner config info.- redis - (optional) configures redis server connection, e.g. redis://example.com:1234?redis_option=value&redis_option=value
Single index concept
- Internally, all [X,Y] coordinates are converted to a single integer, with values 0..(4^zoom-1). This index is constructed by taking bits of both X and Y coordinates one bit at a time, thus every odd bit of the index represents the X, and every even bit represents the Y coordinates.
This allows us to easily treat the whole tile space as one linear space, yet provides for a convenient way to calculate other zoom levels. For example, by simply dividing the index by 4, we get the index of the tile that includes current tile with zoom-1.
Monitoring Jobs
It is highly recomended, although not mandatory, to have an extra instance of the Tilerator running with the uiOnly setting in the config.
This way if Tilerator can be stopped and the pending jobs rearranged. Without the uiOnly instance, you will always be changing the queue
while jobs are running. To configure the uiOnly instance, make a copy of the Tilerator config, set uiOnly to true and change the port number.
To see the currently running jobs, navigate to http://localhost:6534/
(nicer interface) or http://localhost:6534/raw
(internal data).
Job auto-balancing
Tilerator will auto-rebalance jobs if it detects
Adding jobs
Jobs can be scheduled via a POST request. The best way is to use /static/admin.html from the Kartotherian deployment, and point it to Tilerator instance. Or you can make direct calls with Chrome Postman extension or similar.
The most basic call to generate all tiles of zoom level 3, using gen
source to produce tiles, and store them in the store
source. This job will be executed by one worker, without any multitasking.
http://localhost:6534/add?generatorId=gen&storageId=store&zoom=3
Job Parameters
generatorId
- required source ID, as defined in the sources configuration. Tiles from this source will be read.storageId
- required source ID, as defined in the sources configuration. Tiles will be written to this source.zoom
- zoom level to processparts
- break the job into NNN independent jobs, allowing it to run in multiple workers and/or machinesidxFrom
- the starting tile index (inclusive, 0 by default)idxBefore
- generate tiles until this index (non-inclusive, 4^zoom by default)x
andy
- generate just one tile at these coordinates. Cannot be used withidxFrom
oridxBefore
. Coordinates are based on the Slippy Map Tile Names system. These can be generated easily with Geofabrik's calculator.deleteEmpty
- if true, any non-generated tile (e.g. empty or solid) will be explicitly deleted from the storage (optional, false by default)keepJob
- if true, the job will not be automatically removed from the que once it successfully completes
Pyramid mode
Specifying fromZoom
and beforeZoom
enables pyramid mode. This mode tells Tilerator to generate more than
one zoom level with one request. Given a tip of the pyramid - a tile at a given zoom level zoom
,
the pyramid will contain all the tiles under the given tile (higher zooms), and all the tiles that contain the given tile (lower zooms).
So a zoom+1
will be the 4 tiles corresponding to the tip tile, and zoom+2
will have the 16 tiles, etc.
For all zoom levels lower than zoom
, Tilerator will add just one tile per zoom
that contains the tip tile.
Tilerator will only generate the range of zooms requested, which does not have to contain the zoom
(a cross-section of the piramid)
The base zoom may contain more than just one tile or even a whole zoom level. Use idxFrom & idxBefore to specify the tile range, or (x,y) for just one tile.
This feature could be useful for the tile invalidation. For example, a user edited a tile at Z=16, and the system automatically scheduled tile refresh at Z=10..17)
zoom
- the zoom level of the pyramid's tipfromZoom
- zoom level at which to start generation (inclusive)beforeZoom
- zoom level to end tile generation (exclusive)
Parsing updates file
Specifying filepath
parameter will load that file from the localhost and append its content to the queue. The file
must already exist on the server. Alternatively, you can specify expdirpath
, statefile
and expmask
parameters,
which would parse all files stored at the expdirpath
that match regular expression expmask
, and whose string value
is greater than value stored in the file statefile
(filename only). Once parsing is complete, the last filename
will be stored in the statefile, overriding previous content.
Eeach expiration file must be in the format created by osm2pgsql, and all lines must be of the same zoom level:
zoom/x_coordinate/y_coordinate
Tilerator will convert this file into a list of indexes, sort/deduplicate them, and schedule all the needed jobs.
Use fromZoom
and beforeZoom
to invalidate more than just the zoom level given in the file. Note that if the zoom range is
given, only those zooms will be regenerated. So if the zoom level in the file is not in fromZoom <= zoom < beforeZoom
,
that zoom level will not be regenerated. If you have a file has already been converted to the internal format of one index per line,
and has no zoom, the zoom can be supplied via the fileZoomOverride
(only works with filepath
param).
NOTE: Please remember that Tilerator is an admin tool, and should not be accessible from the web
Job Filters
Sometimes you may wish to generate only those tiles that satisfy a certain condition.
If any of these parameters are set, Tilerator will check if a specific tile exists before attempting to regenerate it.
sourceId
- which source to use for filtering. Will usestorageId
by default.checkZoom
- only generate tiles if the corresponding tile exists at zoom levelcheckZoom
. By default, if any other filter values are set, it uses job's zoom level. If negative, means use job's zoom level minus the value.dateBefore
- only generate tile if the tile in storage was generated before given datedateFrom
- only generate tile if the tile in storage was generated after the given datebiggerThan
- only generate tile if the tile in storage is bigger than a given size (compressed)smallerThan
- only generate tile if the tile in storage is smaller than a given size (compressed)missing
- if this is set to true, and other filters are not set, gets all the missing tiles. If other filters are set, generates the missing tiles plus the ones that match the filter. For example, to regenerate missing and small tiles, set thesmallerThan
andmissing
parameters.
Currently /add/
supports up to two filters. Specify the second filter by adding 2
at the end of each filter parameter.
If two filters are given, only tiles that satisfy both filters will be generated.
Queue cleanup and rebalancing
At times, if a job crashes, or the Tilerator is killed by the admin, it will remain in the "active" queue without being worked on,
and its updated
timestamp will stay the same. These jobs have to be moved back to the inactive
queue.
In the future it might be possible to fix this automaticaly, but for now, there is a "clean" POST request:
http://localhost:6534/cleanup
Which by default moves all jobs from active
to inactive
if they haven't been updated for the past 60 minutes.
To process just one job, specify job ID after cleanup:
http://localhost:6534/cleanup/123
Alternativelly, the originating queue and the number of minutes can be specified as the first two value after the cleanup.
This will move all jobs from the failed
queue into inactive
if they haven't been updated in the last 15 minutes:
http://localhost:6534/cleanup/failed/15
cleanup accept these parameters:
- updateSources - if true, will update the sources of all the matching jobs
Copying source info
This command performs copying of the source info
object from source to destination (immediate, not queued)
http://localhost:6534/setinfo/source/destination
Optionally you can specify the ?tiles=
parameter to update it:
http://localhost:6534/setinfo/gen/c?tiles=http://.../osm/{z}/{x}/{y}.pbf
Viewing and dynamically changes sources
You may view currently configured sources by browsing to the /sources
URL (GET)
http://localhost:6534/sources
Sometimes a temporary source is needed for the generation. Tilerator allows dynamic non-permanent sources to be added
via the /sources
command as well, by POSTing the new YAML configuration to the same URL, in the same format as in the sources file.
Note that the operation will add sources to the existing ones, overriding the ones with the same name. Variables cannot be changed.
Viewing defined variables
http://localhost:6534/variables
This URL will show the list of defined variables
tileshell
shell script
Using the Running script/tileshell.js
is similar to making a single POST request to the tilerator admin interface.
A new job can be added by supplying -j.jobparam value
parameters:
node scripts/tileshell.js --config config.yaml -j.fromZoom 0 -j.beforeZoom 5 -j.generatorId gen -j.storageId v5 -j.deleteEmpty
The tool may also be used to export a list of tiles that would be generated. This way one may export existing tiles, or tiles that match a certain filtering criteria:
node scripts/tileshell.js --config config.yaml -j.generatorId v5 -j.zoom 3 --dumptiles listOfZoom3TilesInV5.txt
tileshell
Generate .mbtiles using Running script/tileshell.js
with --writeMbtiles <path>
option will store tiles in a local .mbtiles file.
Note that jobs are processed synchronously.
For example to generate a file with all tiles in Ivory Coast (zoom 0 to 13):
# All tiles on the first zoom levels (0 to 4)
node scripts/tileshell.js --config /etc/tilerator/config.yaml --source /etc/tilerator/sources.yaml -j.fromZoom 0 -j.beforeZoom 5 -j.generatorId ozgen-lite --writeMbtiles ivorycoast.mbtiles
# All tiles in Ivory Coast (zooms 5 to 13)
node scripts/tileshell.js --config /etc/tilerator/config.yaml --source /etc/tilerator/sources.yaml -j.zoom 5 -j.x 15 -j.y 15 -j.fromZoom 5 -j.beforeZoom 14 -j.generatorId ozgen-lite --writeMbtiles ivorycoast.mbtiles