[BUG] Error when building planet tiles: "Protocol message contained an invalid tag (zero)"

[michaelkirk]

Describe the bug

I recently updated planetiler from 0.5.0 to 0.6.0 and am now hitting this error when building the v1.26 daylight map planet file using your docker container.

The planetiler-0.5.0 container successfully builds this same v1.26 planet file.

I've been able to complete other regional builds with planetiler-0.6.0, e.g. Seattle but not the big planet build.

I don't know if it's an issue with planetiler or something wrong with this OSM planet extract, or something unique to the daylight map planet distribution.

+planetiler-build-mbtiles *failed* | 0:20:32 INF [osm_pass1:process] - Finished nodes: 2,543,323,223 (6.4M/s) in 6m33s cpu:47m18s gc:1s avg:7.2                                                                       
+planetiler-build-mbtiles *failed* | Worker osm_pass1_process-1 died                                                                                                                                                  
+planetiler-build-mbtiles *failed* | Exception in thread "main" com.onthegomap.planetiler.util.Exceptions$FatalPlanetilerException: java.util.concurrent.ExecutionException: java.io.UncheckedIOException: com.google.
protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
+planetiler-build-mbtiles *failed* |    at com.onthegomap.planetiler.util.Exceptions.throwFatalException(Exceptions.java:29)                                                                                          
+planetiler-build-mbtiles *failed* |    at com.onthegomap.planetiler.stats.ProgressLoggers.await(ProgressLoggers.java:367)                                                                                            
+planetiler-build-mbtiles *failed* |    at com.onthegomap.planetiler.stats.ProgressLoggers.awaitAndLog(ProgressLoggers.java:352)                                                                                      
+planetiler-build-mbtiles *failed* |    at com.onthegomap.planetiler.reader.osm.OsmReader.pass1(OsmReader.java:221)                                                                                                   
+planetiler-build-mbtiles *failed* |    at com.onthegomap.planetiler.Planetiler.lambda$addOsmSource$0(Planetiler.java:199)                                                                                            
+planetiler-build-mbtiles *failed* |    at com.onthegomap.planetiler.Planetiler.lambda$ifSourceUsed$9(Planetiler.java:841)                                                                                            
+planetiler-build-mbtiles *failed* |    at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:720)                                                                                                              
+planetiler-build-mbtiles *failed* |    at org.openmaptiles.OpenMapTilesMain.run(OpenMapTilesMain.java:54)                                                                                                            
+planetiler-build-mbtiles *failed* |    at org.openmaptiles.OpenMapTilesMain.main(OpenMapTilesMain.java:14)                                                                                                           
+planetiler-build-mbtiles *failed* |    at com.onthegomap.planetiler.Main.main(Main.java:84)                                                                                                                          
+planetiler-build-mbtiles *failed* | Caused by: java.util.concurrent.ExecutionException: java.io.UncheckedIOException: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (
zero).
+planetiler-build-mbtiles *failed* |    at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)                                                                                                 
+planetiler-build-mbtiles *failed* |    at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)                                                                                                       
+planetiler-build-mbtiles *failed* |    at com.onthegomap.planetiler.stats.ProgressLoggers.await(ProgressLoggers.java:361)                                                                                            
+planetiler-build-mbtiles *failed* |    ... 8 more                                                         
+planetiler-build-mbtiles *failed* | Caused by: java.io.UncheckedIOException: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
+planetiler-build-mbtiles *failed* |    at com.onthegomap.planetiler.reader.osm.OsmInputFile$LazyReader.forEachBlock(OsmInputFile.java:213)
+planetiler-build-mbtiles *failed* |    at com.onthegomap.planetiler.reader.osm.OsmReader.lambda$pass1$2(OsmReader.java:192)                                                                                          
+planetiler-build-mbtiles *failed* |    at com.onthegomap.planetiler.worker.WorkerPipeline$Empty.lambda$fromGenerator$0(WorkerPipeline.java:170)
+planetiler-build-mbtiles *failed* |    at com.onthegomap.planetiler.worker.Worker.lambda$new$0(Worker.java:41)             

To Reproduce
Steps to reproduce the behavior:

mkdir data
# download lake centerline etc into data
# Download planet file (Warning 71GB file)
curl https://daylight-map-distribution.s3.us-west-1.amazonaws.com/release/v1.26/planet-v1.26.osm.pbf > data/planet-v1.26.osm.pbf

# build tiles
docker run  -v=data:/data ghcr.io/onthegomap/planetiler:0.6.0 --force --osm_path=/data/planet-v1.26.osm.pbf

Expected behavior

I expected the build to succeed without crashing.

Screenshots

n/a

Environment (please complete the following information):

• Hardware: I'm running your docker container on an x86_64 Debian VM
• OS: whatever is in your docker container
• Java version and distribution: whatever is in your docker container
• Maven version: whatever is in your docker container

[msbarry]

I wonder if the download got corrupted somehow? When I run like this (on an r7g.metal instance) it reads the pbf file fine:

docker run  -v=$(pwd)/data:/data ghcr.io/onthegomap/planetiler:0.6.0 --area=planet --osm-url=https://daylight-map-distribution.s3.us-west-1.amazonaws.com/release/v1.26/planet-v1.26.osm.pbf  --bounds=planet --nodemap-type=array --storage=mmap --force --download-threads=100 --download-chunk-size-mb=100 --download

[michaelkirk]

Sorry for the incredibly slow feedback loop, but I haven't seen this bug re-appear. You might be right that it was a corrupted download.