Rail links omitted
JWJoubert opened this issue · 16 comments
Good morning, I am unsuccessfully trying to parse a mapped public transport network from OSM. The Osm2TransitSchedule seems to honour the necessary keys and tags, and we made sure that the tagging in OSM is done correctly. The problem seems to actually be the multimodal network created using Osm2MultimodalNetwork.
Many rail/train links seem to be omitted but I cannot find the issue. In OSM the railway tracks are well-formed and fully connected (as far as I can ascertain). I also looked into the log file to see if any specific tags were omitted:
...
2021-06-28T02:38:05,521 INFO OsmMultimodalNetworkConverter:284 The following railway-types had no defaults set and were thus NOT converted:
2021-06-28T02:38:05,521 INFO OsmMultimodalNetworkConverter:286 - "funicular"
2021-06-28T02:38:05,521 INFO OsmMultimodalNetworkConverter:286 - "preserved"
2021-06-28T02:38:05,521 INFO OsmMultimodalNetworkConverter:286 - "station"
2021-06-28T02:38:05,521 INFO OsmMultimodalNetworkConverter:286 - "miniature"
2021-06-28T02:38:05,521 INFO OsmMultimodalNetworkConverter:286 - "abandoned"
2021-06-28T02:38:05,521 INFO OsmMultimodalNetworkConverter:286 - "disused"
2021-06-28T02:38:05,521 INFO OsmMultimodalNetworkConverter:286 - "platform"
2021-06-28T02:38:05,521 INFO OsmMultimodalNetworkConverter:286 - "razed"
2021-06-28T02:38:05,521 INFO OsmMultimodalNetworkConverter:295 = end of conversion statistics ====================
Can you maybe point me to the area where I can get a better idea of how nodes/ways are filtered out.
I am using the default config:
OsmConverterConfigGroup config = OsmConverterConfigGroup.createDefaultConfig();
config.setOsmFile(osmFile);
config.setOutputCoordinateSystem(Utils.CRS_LO_29);
config.setOutputNetworkFile(networkFile);
Osm2MultimodalNetwork.run(config);
without making any specific changes. So I do not know if it might just be a config-setting that requires more specific attention. I did not pick up anything specific from the Wiki page.
Hi Johan, can you send the OSM id of a link that is not converted, to have an example?
I'm not sure if I understand correctly, there are railway:rail links missing, or the ones from the list above are missing?
Usually, the modes to convert are defined in the converter config (usually I first run org.matsim.pt2matsim.run.CreateDefaultOsmConfig, modify the config, and then run the converter, but this should also be possible in code by adding the right ParameterSet):
<parameterset type="wayDefaultParams" >
<param name="allowedTransportModes" value="rail" />
<param name="freespeed" value="44.44444444444444" />
<param name="freespeedFactor" value="1.0" />
<param name="laneCapacity" value="9999.0" />
<param name="lanes" value="1.0" />
<param name="oneway" value="false" />
<param name="osmKey" value="railway" />
<param name="osmValue" value="rail" />
</parameterset>So here we make sure that railway:rail links are converted, by default, there is also tram and light_rail.
Maybe it could be possible that these parameter sets are not added by default if you create the config in code (by I doubt that), but it may be worth a try to also generate the config and then run the converter.
More complex explanation: Usually, the converter will only keep consistent and connected sub-networks. For instance, you find
<parameterset type="routableSubnetwork" >
<param name="allowedTransportModes" value="car" />
<param name="subnetworkMode" value="car" />
</parameterset>
<parameterset type="routableSubnetwork" >
<param name="allowedTransportModes" value="bus,car" />
<param name="subnetworkMode" value="bus" />
</parameterset>in the config, which means: In the multimodal network, find the biggest cluster of links with the converted mode car (see the parameter set above for "highway") and include it in the final network with "car" as the MATSim transport mode. Also, find that largest cluster where links have either the car or bus mode, and keep those links, with "bus" as the MATSim transport mode. This way we have a consistent network for car, and a consistent network for bus, and they are encoded on the same links in the final MATSim networks where there are overlaps. Basically, this means we have a baseline car network and on top there may be links that are bus-only streets.
Now railways are usually not well-connected in OSM, so using this sub-network feature would result in deleting a lot of links. That's why there is the flag:
<param name="keepWaysWithPublicTransit" value="true" />The idea is to keep public transport links, even if they don't belong to a consistent network (as the tracks are often independent).
The corresponding code is OsmMultimodalNetworkConverter::initPt where the filter is created. Currently, there are many things hardcoded as criteria, probably could make sense to make this more flexible. But this could now be where your links are filtered out as well. Note that this is mainly based on the relations that the links belong to in OSM.
I just checked, for France, the railways links seems to be kept in the converter, because the relations they belong to usually have a route=railway tag. Maybe this is not the case in South Africa?
can you send the OSM id of a link that is not converted, to have an example?
Sure, thanks @sebhoerl. Here is a way that is not included but it has the correct (in my view) tags: railway=rail. It is part of a relation for a train route.
While this way is captured in the same way, yet it does appear in the multimodal network.
Now railways are usually not well-connected in OSM, so using this sub-network feature would result in deleting a lot of links. That's why there is the flag:
I suspected that much, but my flag is set correctly, true, as you've indicated.
Here is a train route relation where the tag is different to what you suggested, @sebhoerl. The route is set as route=train and not route=rail but it is still included.
...but it may be worth a try to also generate the config and then run the converter.
I will try that.
I will try that.
I think it is not necessary, I'm pretty sure it is a filtering issue after writing all of this above.
All the criteria for keeping a PT link are here:
So route=rail is included. However, often in your examples we have route=railway which seems not to be part of the current filters. On the other hand, the examples you showed all seem to be covered at least by route=train or route=rail by at least one relation.
The detailed way would be to go to
How do you generate the input data for the converter? Maybe by cutting out a certain region some other tools has removed relations that go beyond the cut-out region? May also be an explanation.
Thanks, @sebhoerl
put an if there or a breakpoint listening to the relevant way ID that gets filtered out and then checking why it gets filtered out.
Will do.
How do you generate the input data for the converter? Maybe by cutting out a certain region some other tools has removed relations that go beyond the cut-out region? May also be an explanation.
It is a server script that now runs nightly. I use osmosis and cut all features using a polygon. Typically something like:
osmosis/bin/osmosis --read-pbf file="input.pbf" --bounding-polygon file="polygon.file.poly" completeWays=yes --write-pbf file="output.pbf"
I would think that the completeWays=yes would have solved that. But let me check, maybe there is a similar flag for relations.
There indeed is a completeRelation flag. Let me add that too and see if it makes a difference.
... I doubt this will have much of an effect as we have the same issue in Cape Town, and there the polygon cannot cut the boundary: it is wholly contained in the area.
So, quick update, @sebhoerl and @polettif... I tried the conversion with the default config file, and not the config object using
OsmConverterConfigGroup config = OsmConverterConfigGroup.createDefaultConfig();
... it works, I tried the breakpoint, but the code never paused there... the plot thickens. Now I need to understand what the difference between the two is. I would have reasoned they are the same.
I tried the conversion with the default config file, and not the
configobject using ... it works
Just to make a note here: Maybe this is a similar issue to #152 where code-based and file-based config objects don't behave the same (i.e. some params are not written/read).
No, they only fixed this for PublicTransitMappingConfigGroup, not the OsmConverter. We didn't look at all config/file related code in that issue (there are probably exhaustive tests necessary for that).
Hi, @polettif. I am still trying to figure out why pt2matsim keeps ignoring OSM (train) routes that I cannot find fault with in terms of the tagging consistency. I think I may have gotten closer to an answer. Firstly, according to the tagging guidelines for route_master it seems that while train is allowed, it is, unfortunately, missing in the code. Adding it solves a lot of issues.
Secondly, there are sometimes nodes that form part of a relation that does not exist in the facilities. In my view, these should just be ignored. Yet, the code currently is
TransitStopFacility transitStopFacility = transitSchedule.getFacilities().get(id);
if(transitStopFacility == null) {
return null;
}
// create transitRouteStop
TransitRouteStop newRouteStop = factory.createTransitRouteStop(transitStopFacility, 0.0, 0.0);
stopSequenceForward.add(newRouteStop);
which ignores any other (potential) stops, rendering the route null. The following is proposed as less restrictive and also solves a lot of our problems.
if(transitStopFacility != null) {
// create transitRouteStop
TransitRouteStop newRouteStop = factory.createTransitRouteStop(transitStopFacility, 0.0, 0.0);
stopSequenceForward.add(newRouteStop);
}
I can create a pull request if you approve of the changes.
Firstly, according to the tagging guidelines for
route_masterit seems that whiletrainis allowed, it is, unfortunately, missing in the code. Adding it solves a lot of issues.
Ah I see. Don't know why train's not added there, I guess this was for road-based (or adjacent) pt vehicles.
Secondly, there are sometimes nodes that form part of a relation that does not exist in the facilities. In my view, these should just be ignored.
Yes, ignoring the route is overly strict. Maybe a warning could be issued?
I can create a pull request if you approve of the changes.
Yes, they look good to me, a PR would be very much appreciated!
I just now fully realized you're talking about OsmTransitScheduleConverter and not OsmMultimodalNetworkConverter. The osm schedule converter is more a proof of concept or at least I haven't really tested it.