Problem with building OSM Europe with inner-outer-geoms
hannahbast opened this issue · 10 comments
The run ended in the middle of constructing the non-reduced DAG, without an error message or anything. The following part of the log makes it looks as if the program terminated normally, but of course it didn't. What's happening here?
bast@lena:osm-europe$ qlever download-data
This is the "qlever" script, call without argument for help
Executing "download-data":
wget -nc -O osm-europe.pbf https://download.geofabrik.de/europe-latest.osm.pbf; time ( /local/data/osm2rdf/build/apps/osm2rdf osm-europe.pbf -o osm-europe.ttl --cache . --store-locations-on-disk --split-tag-key-by-semicolon ref | tee osm-europe.osm2rdf-log.txt )
Downloading data using DOWNLOAD_CMD from Qleverfile ...
File ‘osm-europe.pbf’ already there; not retrieving.
[2022-06-26 05:40:30.901] osm2rdf :: 75f47d4-dirty :: BEGIN
Config
--- I/O ---
Input: "osm-europe.pbf"
Output: "osm-europe.ttl.bz2"
Output format: qlever
Cache: "/local/data/qlever/qlever-indices/osm-europe/."
--- Facts ---
Simplifying WKT
Simplifying wkt geometries with factor: 5
Dumping WKT with precision: 7
Tag-Keys split by semicolon:
ref
--- Contains ---
--- Miscellaneous ---
Storing locations osmium locations on disk
--- OpenMP ---
Max Threads: 16
[2022-06-26 05:40:30.902] Free ram: 0.783966G/125.803G
[2022-06-26 05:40:30.903] OSM Pass 1 ... (Relations for areas)
[2022-06-26 05:41:27.682] ... done ] 0%
[======================================================================] 100%
[2022-06-26 05:41:27.725] OSM Pass 2 ... (dump)
[======================================================================] 100%
[2022-06-26 07:16:33.199] ... done reading (libosmium) and converting (libosmium -> osm2rdf)
[2022-06-26 07:16:33.199] areas seen:258692004 dumped: 258692004 geometry: 258692004
nodes seen:2984713721 dumped: 107927718 geometry: 107927718
relations seen:6183874 dumped: 6183296 geometry: 0
ways seen:361552080 dumped: 354195428 geometry: 354195428
[2022-06-26 07:16:33.199] Calculating contains relation ...
[2022-06-26 07:16:33.206] Sorting 6000770 areas ...
[2022-06-26 07:16:34.358] ... done
[2022-06-26 07:16:34.358] Packing area r-tree with 6000770 entries ...
[2022-06-26 07:16:41.242] ... done
[2022-06-26 07:16:42.474] Generating non-reduced DAG from 6000770 areas ...
[=====================================> ] 68% [4136030/6000770]
real 270m46.969s
user 3934m0.286s
sys 218m14.121s
Just testing "creating an issue from project"
I was finally able to reproduce this in gdb. As suspected, it is a stack overflow in
osm2rdf/src/util/DirectedGraph.cpp
Line 53 in 03cc182
This is a recursive function which produces all successors for a given node in a DAG. As the graph is assumed to be acyclic, nodes are not marked as visited. For some reason, a cycle is introduced for europe-latest.osm, and this recursion never ends. The code that builds the DAG has an explicit check if two areas A and B are equivalent (https://github.com/ad-freiburg/osm2rdf/blob/inner-outer-geoms/src/osm/GeometryHandler.cpp#L589), and if so, does not insert any edge between the corresponding nodes in the DAG. This is to avoid adding two edges A -> B and B -> A and thus introducing a cycle. If for some reason (floating point inaccuracies?) it holds that both A in B and B in A, but A != B, a cycle will be inserted into the graph. I strongly suspect that is what happens here.
After talking with Axel, we have two suspicions why this only happens on europe-latest.osm, and not planet.osm.
1.) Different data sources. europe-latest.osm is from Geofabrik and has been touched by at least one filtering tool, possibly adding slight floating point inaccuracies, while planet.osm is a raw export from OSM.
2.) The insertion order might be different, and/or areas already present in the DAG might prevent adding the erroneous edge.
The obvious solution to check whether an edge B -> A is present in the graph if A -> B is about to be added will not work, as it is still possible to produce a cycle A -> B -> C -> A. We should instead replace the equivalence check (boost::geometry::equal(a, b)) in https://github.com/ad-freiburg/osm2rdf/blob/inner-outer-geoms/src/osm/GeometryHandler.cpp#L589 by boost::geometry::covered_by(a, b) && boost::geometry::covered_by(b, a), thus using the exact same algorithm for checking equivalency as for checking whether the areas are contained in each other.
I will test this today on ob.
Now running on ob, log goes to /local/data/brosip/osm2rdf.log.
@patrickbr @lehmann-4178656ch Wow, thanks for looking into this + looking forward to the outcome!
With e0ae455 the DAG build for europe-latest.osm was successful and took 4 hours and 37 minutes. Currently building the node-in-area relations.
Quick update: run with full geometries on europe-latest.osm has finished and took 4.5 days. Run with --approximate-spatial-relations has been started, log goes to /local/data/brosip/osm2rdf-approx.log.
Did the approx run crash or stall? The log file was last updated on 12.07.2022 at 13:45.
No, we stopped it because the weekly osm2rdf update was running at the same time, which would've resulted in incomparable running times. I restarted it a few minutes ago :)
I restarted it a few minutes ago :)
Which is not that great for comparability as the weekly updates will start tommorrow evening.
My hope was that the run will be finished until tomorrow, but you are probably right that it won't :) wolga seems to be idling, I will start it there.