[BUG] tiles missing >= zoom 7

[hyperknot]

Describe the bug
In the generated planet.mbtiles, not all tiles are present.

To Reproduce

1. Run current full planet gen.
2. Open sqlite3 planet.mbtiles

sqlite> select count(*) from tiles_shallow where zoom_level = 1;
4
sqlite> select count(*) from tiles_shallow where zoom_level = 2;
16
sqlite> select count(*) from tiles_shallow where zoom_level = 3;
64
sqlite> select count(*) from tiles_shallow where zoom_level = 4;
256
sqlite> select count(*) from tiles_shallow where zoom_level = 5;
1024
sqlite> select count(*) from tiles_shallow where zoom_level = 6;
4096
sqlite> select count(*) from tiles_shallow where zoom_level = 7;
13986
sqlite> select count(*) from tiles_shallow where zoom_level = 8;
55101
sqlite> select count(*) from tiles_shallow where zoom_level = 9;
217800
sqlite> select count(*) from tiles_shallow where zoom_level = 10;
853834

As you can see, it goes perfectly until zoom 6, but from zoom 7 some tiles are missing.

Expected behavior
All tiles should be generated.

Here is the same from a reference openmaptiles implementation (the super slow one). Not a single tile missing.

sqlite> select count(*) from map where zoom_level = 7;
16384
sqlite> select count(*) from map where zoom_level = 8;
65536
sqlite> select count(*) from map where zoom_level = 9;
262144
sqlite> select count(*) from map where zoom_level = 10;
1048576
sqlite> select count(*) from map where zoom_level = 11;
4194304
sqlite> select count(*) from map where zoom_level = 12;
16777216
sqlite> select count(*) from map where zoom_level = 13;
67108864
sqlite> select count(*) from map where zoom_level = 14;
268435456

Environment (please complete the following information):

• Hardware: Linux server
• OS: Ubuntu 22
• Java version and distribution:

java -version
openjdk version "17.0.9" 2023-10-17
OpenJDK Runtime Environment (build 17.0.9+9-Ubuntu-122.04)
OpenJDK 64-Bit Server VM (build 17.0.9+9-Ubuntu-122.04, mixed mode, sharing)

• Maven version: -

[hyperknot]

It's interesting that these number are more or less consistent between a run today and one done 2 months ago.

sqlite> select count(*) from tiles_shallow where zoom_level = 1;
4
sqlite> select count(*) from tiles_shallow where zoom_level = 2;
16
sqlite> select count(*) from tiles_shallow where zoom_level = 3;
64
sqlite> select count(*) from tiles_shallow where zoom_level = 4;
256
sqlite> select count(*) from tiles_shallow where zoom_level = 5;
1024
sqlite> select count(*) from tiles_shallow where zoom_level = 6;
4096
sqlite> select count(*) from tiles_shallow where zoom_level = 7;
13986
sqlite> select count(*) from tiles_shallow where zoom_level = 8;
55100
sqlite> select count(*) from tiles_shallow where zoom_level = 9;
217779
sqlite> select count(*) from tiles_shallow where zoom_level = 10;
853600

[msbarry]

In the openmaptiles profile, "land" is indicated by absence of any data, and planetiler doesn't empty tiles so the counts dropping starting at z7 are from entire tiles being excluded from the output (probably starting in antarctica).

Also FYI you should upgrade to java 21 - newer versions of planetiler won't work with java 17 anymore.

[hyperknot]

So with a decent client lib it should just work correctly, right? Thanks for the java 21 hint, I didn't see the update 2 month ago.

[pnorman]

In the openmaptiles profile, "land" is indicated by absence of any data

Land and the absence of data would be a zero-byte tile, not the absence of a tile.

[msbarry]

Right, the translation from "no data" to zero-byte tile needs to happen at some point in the stack. A tile server like tileserver-gl will return a 204 response when the tile is missing from the underlying data source. The pmtiles javascript client passes an empty byte array to maplibre when the tile is missing from the index, and maplibre-gl handles a 404 as no data.

[hyperknot]

Do you mean what's missing is a zero byte file? I'll be serving this directly, from the filesystem, without any tile server.
So basically I need to make sure that all 404s are returned as an empty string and 200?

[msbarry]

What client are you be using?

[hyperknot]

I want to make it compatible with every client which works with the reference implementation (for example Mapbox GL JS 1.x, etc.). So I need to do this server side.

[hyperknot]

I think this nginx block should take care of it, but I need a bit more testing:

location /.../ {
    alias ...;
    try_files $uri $uri/ @empty;
}

location @empty {
    default_type text/plain; # TODO
    return 200 "''";
}

[msbarry]

Would return 204; work?

[hyperknot]

I'm not sure how browsers, proxies, CloudFlare, etc. handle 204 requests.
I'd like to replicate the behaviour of the reference implementation

The reference implementation + https://github.com/consbio/mbtileserver returns 200 with content-type: application/x-protobuf

(tested on /14/0/16284.pbf and /14/0/16247.pbf)

[pnorman]

Do you mean what's missing is a zero byte file? I'll be serving this directly, from the filesystem, without any tile server. So basically I need to make sure that all 404s are returned as an empty string and 200?

Directly from the filesystem? If so, then you actually need the 0 byte files. If you're serving it not directly, but with a HTTP server between the tiles and the client, then you could transform missing files into zero byte 2xx responses. You'll just have to be careful about distinguishing between tiles that are missing because they're really 0-byte tiles, tiles that are missing because they're invalid (e.g. 0/1/2 or 0/1/-1), URLs that aren't tiles at all, and tiles outside your zoom range and bounding box.

returns 200 with content-type: application/x-protobuf

As per the specification, the MIME type SHOULD be application/vnd.mapbox-vector-tile

[hyperknot]

Yes, I meant that nginx will be serving them directly from the filesystem, without any tile server running. Thank you for the explanation, I understand it now and will update the MIME type.