Conversion HERE Shapefile > ogr2osm > bz2 > Navit .bin using Maptool
peterbmckinley opened this issue · 15 comments
Dear Mr Norman,
I'm on a mission to convert a Shapefile supplied by Nokia HERE (Microsoft) for use in the open source Navit satnav application. Navit have their own Planet Extractor website where users can download OSM derived .bin files for use with Navit. I have also successfully used Maptool to convert both OSM XML and OSM Protobuf files from Geofabrik to the .bin format required by Navit. However the OSM map for Northern Ireland is missing a lot of crucial data, for example most speed limit information etc.
The steps I have taken so far (Lubuntu 16.04.1 LTS 64bit, 8GB Physical + 8GB swapfile) have been as follows:
1: Convert the HERE Shapefile folder to a single .osm file using your ogr2osm utility: SUCCESS
2: Compress .osm file to bz2 as required by Maptool: SUCCESS
3: Convert osm.bz2 file to Navit friendly .bin file using Maptool: FAIL
The message I get is as follows:
peter@peter-desktop:~$ cd ogr2osm
peter@peter-desktop:~/ogr2osm$ bzcat hereni.osm.bz2 | maptool hereni.bin
PROGRESS: Phase 1: reading input data 0:00 0 MB
PROGRESS1: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 0:00 0 MB
INFO: Nodes out of sequence (new 4294967293 vs old 4294967294), adding hash
INFO: Ways out of sequence (new -55 vs old -1), adding hash
FATAL: wrong line in input data (does not start with '<'): >
This does not look like a valid OSM file.
Note that maptool can only process OSM files without wrapped or empty lines.
peter@peter-desktop:~/ogr2osm$ "
As I said, I've used Maptool to convert OSM files (osm.bz2) from Geofabrik into the required .bin files without any problems, and they work, but it's not happy with the one I created myself from a HERE shapefile using ogr2osm. The process of using ogr2osm went smoothly without any errors.
Any interpretation of the error message above will be most gratefully received!!!
Regards,
Peter McKinley
Here is a screenshot of all the Maptool options:
Note that maptool can only process OSM files without wrapped or empty lines.
This sounds like a badly broken XML parser. You could try passing the ogr2osm XML through xmllint to reformat it.
Hi Paul,
I appreciate your input. The process of running the .osm through xmllint completed without issue, however when I zipped it and ran it through maptool the result was exactly as described above :( I know Maptool isnt your department, but it is in the Ubuntu repository (sudo apt-get install maptool). Have i come to the end of the road (pun intended :)) or can you suggest any other switches in xmllint or maptool that I could try?
Or something else entirely!
Thanks again.
Regards,
Peter
try adding --positive-id --add-version --add-timestamp. That should produce an XML file the same as the API would, and if that doesn't fix it, try reducing it to a smaller testcase and reporting it to maptool
Apologies, add these in which step? ogr2osm or xmllint?
Ignore that question, it's ogr2osm :) Trying it now. It'll take a while, the source shapefile is 303MB.
Peter
Incidentally, prior to running ogr2osm I included 2 related .csv files which had previously been outside the main shapefile directory. I placed them in among the .prj, .dbf etc. files. Was that a mistake?
Peter
Shouldn't matter.
I recommend cutting out a small part of the shapefile with ogr2ogr and working with that to test.
Hi Paul,
You may wish to read the last paragraph below first!
No joy I'm afraid, adding those options only doubled the resulting .osm file size but the maptool result was the same, and when I tried running the new .osm file through xmllint the pc froze after 10 minutes. Tried again after a reboot but exactly the same error and freezing.
Going back to basics I did a vanilla install of Lubuntu (16.04.1 64bit 8GB+8GB swap) and did the following testing using freshly downloaded Isle of Man files from Geofabrik:
1: isle-of-man-latest.osm.bz2 (3.2MB) converted to .bin file using Maptool: SUCCESS, loads into Navit
2: isle-of-man-latest.osm.pbf (2.2MB) converted to .bin file using Maptool: SUCCESS, loads into Navit
3: isle-of-man-latest-free.shp.zip (8.5MB after unzipping) converted to .osm using ogr2osm then resulting .osm file converted to .bin file using Maptool: FAIL (error as above, " FATAL: wrong line in input data (does not start with '<'): > "
I tried step 3 using all 3 possible methods of converting .osm files listed on
http://wiki.navit-project.org/index.php/OpenStreetMap
namely:
a: cat my_OSM_map.osm | maptool my_Navit_map.bin
b: maptool -i my_OSM_map.osm my_Navit_map.bin
c, after zipping to bz2: bzcat my_OSM_map.osm.bz2 | maptool my_Navit_map.bin
all with the identical error result above.
In an effort to isolate the problem, I exported a 2.5MB .osm of my home town (Bangor, Northern Ireland) directly from https://www.openstreetmap.org/export. Amazingly I was able to convert this to a Navit friendly .bin file using both maptool methods a and b above, and they both worked in Navit!! Therefore I have proved that maptool works with OSM XML in bz2 form and OSM Protobuf data in .pbf form, both from Geofabrik, and now also OSM XML in native .osm form directly from OpenStreetMap.
However for some reason maptool doesn't like your ogr2osm converted shapefiles, either my HERE shapefile or an unzipped shapefile from Geofabrik. The conversion runs without errors, and yet maptool complains with the error as listed above, namely:
peter@peter-Lubuntu-64bit:~$ maptool -i /home/peter/ogr2osm/isle-of-man-latest-free.osm isle-of-man-latest.bin
PROGRESS: Phase 1: reading input data 0:00 0 MB
PROGRESS1: Processed 0 nodes (0 out) 0 ways 0 relations 0 tiles 0:00 0 MB
INFO: Nodes out of sequence (new 4294967293 vs old 4294967294), adding hash
INFO: Ways out of sequence (new -35 vs old -1), adding hash
FATAL: wrong line in input data (does not start with '<'): >
This does not look like a valid OSM file.
Note that maptool can only process OSM files without wrapped or empty lines.
peter@peter-Lubuntu-64bit:~$
If you think you know where the problem may lie I am happy to consider paying you for your time to fix it if you have a Paypal account. I am extemely grateful for your willingness to engage thus far. I feel ogr2osm is my best if not my only hope for what I'm trying to do.
Best regards,
Peter
By the way for completeness, I also created a new Isle of Man .osm from my unzipped Geofabrik IoM shapefile using ogr2osm this time including --positive-id --add-version --add-timestamp. No change in maptool. Ran this new .osm through xmllint. No change in maptool. There's literally no other permutation I can try! :) Hope you can help. Peter
Can you try opening the ogr2osm XML in a text editor and trimming it down to a test of a few dozen lines?
trimming it down to a test of a few dozen lines?
You'll need to make sure it's valid XML, which means keeping the closing stuff.
For taking on paid work, I am available. I suspect the problem is ultimately with maptool, but replicating exactly what maptool wants might act as a work-around. I haven't worked with the ogr2osm code for a couple years now.
Hi Paul,
Using a sample Isle of Man shapefile from Geofabrik I found that if I removed the "way id" components of your resulting .osm and left only node and relation id's the resulting file would process successfully in maptool and load into Navit. Obviously this leaves a fairly useless map, but what does it tell us about the error?
As I said I was able to process (in maptool) a .osm of my hometown of Bangor from https://www.openstreetmap.org/export and view it in Navit.
The latest sourcecode for maptool can be found here:
https://github.com/navit-gps/navit/releases/tag/v0.5.0
You'll find it in /navit-0.5.0/navit/maptool
If you identify work that needs to be done on either maptool or ogr2osm we can discuss it if you are able to give me a quote.
Best regards,
Peter
Hi Paul,
The .osm file created by ogr2osm included way id lines that were over 3000 columns wide. For some reason maptool was unable to process these. Your advice to run the .osm through xmllint was correct, I just wasn't using xmllint correctly. I wasn't giving xmllint a proper output and also I needed the --pretty 1 argument to actually fix the problem.
Thanks for pointing me in the right direction, sorry if you've spent time thinking about this since your last message.
Best regards,
Peter