How to show all active trains in Berlin on a map?

[creativecreatorormaybenot]

Description

What I want to do essentially is find out both where all trains (U-Bahn, S-Bahn, & Tram) are currently located and ideally also know all of the shapes of the train lines.

I do not care about regional trains and busses for now (although I would assume they would not differ much).

Examples

I found this view in the VBB app that pretty much fulfills the first of the criteria (i.e. knowing the current location of all active trains in Berlin):

It updates the position of all trains in realtime (and also shows the direction they are heading in).

I would ideally want to combine that with the line map in order to get all of the shapes of the train lines:

APIs to use

It seems that this should be possible with the transport.rest APIs (given this is possible in the DB / VBB apps).

Initially, I thought I would have to handle U-Bahn & Tram separately from the S-Bahn (since the S is DB-operated and the U & Tram are BVG-operated). However, given that the VBB app seems to have all of the information, I am thinking I might be able to use just https://v5.vbb.transport.rest/ instead of https://v5.bvg.transport.rest/ and https://v5.db.transport.rest/ separately.

---

I have not yet explored the APIs enough to make an informed decision here and it does seem overwhelming to me as well. I also do not know if https://v0.berlin-gtfs-rt.transport.rest/ would be any use to me.

@derhuerst do you have insights on how to break down this problem to figure out how to (best) do this?

[creativecreatorormaybenot]

When I played around with it initially, I thought I might be able to use the Open API playground to go through the whole scenario, but I was not really successful at getting any information out of it without first knowing what to look for.

My thought was something along the lines of:

1. Get all of the lines with /lines.
2. Then get the individual shapes via /shapes/{id}.
3. And then maybe /radar for getting the positions.

---

This could be done effeciently by only doing 1. and 2. once on my side (maybe updating it every month or so, whenever the lines change).

And after playing around more, I am easily able to get the IDs from the lines endpoint:

[
{"type":"line","id":"17526_400","name":"U9","operator":"796","mode":"train","product":"subway"},
{"type":"line","id":"17525_400","name":"U8","operator":"796","mode":"train","product":"subway"},
{"type":"line","id":"17523_400","name":"U7","operator":"796","mode":"train","product":"subway"},
{"type":"line","id":"17521_400","name":"U6","operator":"796","mode":"train","product":"subway"},
{"type":"line","id":"17518_400","name":"U5","operator":"796","mode":"train","product":"subway"},
{"type":"line","id":"17516_400","name":"U4","operator":"796","mode":"train","product":"subway"},
{"type":"line","id":"17515_400","name":"U3","operator":"796","mode":"train","product":"subway"},
{"type":"line","id":"17514_400","name":"U2","operator":"796","mode":"train","product":"subway"},
{"type":"line","id":"17512_400","name":"U1","operator":"796","mode":"train","product":"subway"}
]

This already seems ideal. About the shapes I will have to see because I also saw https://github.com/derhuerst/vbb-shapes and I am not currently sure how it is connected.

And the realtime fetching is likely the biggest question - also how that works with the 100 request per minutes rate limit.

Edit: so I just realized that shapes are actually not connected to lines but to trips (https://github.com/derhuerst/vbb-trips). Thinking about it, this makes a lot of sense. But I guess it shows that it does make sense to ask this question to get a wholistic idea of how to best solve this before attempting to do it myself and struggling for too long :)

[derhuerst]

It sounds a lot like you're trying to build something like hvv.live for Berlin. A while ago I tried to get in touch with @ubilabs via @aemkei in order to re-use the code, but with no success. Maybe we can re-use the code across cities this time. 🙌

[derhuerst]

I found this view in the VBB app that pretty much fulfills the first of the criteria (i.e. knowing the current location of all active trains in Berlin):

It updates the position of all trains in realtime (and also shows the direction they are heading in).

Note that, as of fall 2021, these are not actual measured realtime positions; The positions are estimated (in a really dumb way, someone should improve upon this!) based on the realtime delay of the trip.

So there's sone truth to that, but it's far from being as accurate as e.g. in Finland. Someone told me that the BVG buses only send a ping whenever they stop at a stop/station, so the current delay is a (very dumb, someone should improve upon this!) estimation.

I assume, but don't know, that for trains, the backend system knows at least the track section it's in and uses that.

[...] get all of the shapes of the train lines:

As you noted below, a line usually has >1 shape, for several reasons:

• Some lines have trips with varying sets of stops for each direction (e.g. U2 in Berlin).
• Vehicles run on different tracks, lanes or even roads when they go back and forth.
• There are sometimes special trips, e.g. those that circumvent construction work in the night, or those that continue until the bus depot.

So it's up to you:

• Do you want any shape for each line?
• Do you want to show the shape of the specific trip? (easy)
• Do you want to show the "canonical" shape(s) of each line? (not super easy, I experimented with this a while ago)

[...] given that the VBB app seems to have all of the information, I am thinking I might be able to use just https://v5.vbb.transport.rest/ [...].

Correct!

I have not yet explored the APIs enough to make an informed decision here and it does seem overwhelming to me as well. I also do not know if https://v0.berlin-gtfs-rt.transport.rest/ would be any use to me.

Yes, the latter helps you in a significant way: It is designed to always give you the position of all trips within the Berlin area at once. It is in the GTFS-Realtime format, which is intended to answer "Which vehicles/trips are running in the network right now?".

With the former, you'd have to determine this by yourself:

• repeatedly find all trips currently running (not that straightforward)
• periodically update each trip's vehicle position

The software behind https://v0.berlin-gtfs-rt.transport.rest/, hafas-gtfs-rt-feed, does exactly that, so you could either

• build this polling logic by yourself (as explained anove), or
• deploy just hafas-monitor-trips and use its data (which is the hafas-client reponse format and thus the same as with v5.{bvg,vbb,db}.transport.rest, or
• deploy hafas-gtfs-rt-feed (you would rather deploy berlin-gtfs-rt-server, which is a thin wrapper around it), or
• just use https://v0.berlin-gtfs-rt.transport.rest/.

The best way to go depends on

• your time: How quickly do you want to get your project working?
• what your goal is: do you want to build this "just" for Berlin? A cool second step might be to build a generic transit live map (like TransitTracker or my more technical & less polished gtfs-rt-inspector), but with the same UX & cool design as hvv.live!
• how important self-hosting & tinkering are for you: the lower you go in the "stack", the more you can tweak

[derhuerst]

[...] I thought I might be able to use the Open API playground to go through the whole scenario, but I was not really successful at getting any information out of it without first knowing what to look for.

What is it that you would have needed? A guide like this one, bit focused on the "what is running right now and where?" use case?

3. And then maybe /radar for getting the positions.

That's the one you're looking for! Underneath, its exactly what the VBB app does in your screenshot, and it's also what hafas-gtfs-rt-server/berlin-gtfs-rt-feed/v0.berlin-gtfs-rt.transport.rest do.

This could be done effeciently by only doing 1. and 2. once on my side (maybe updating it every month or so, whenever the lines change).

Now that I've thought about it, going entirely with static/plan data as the base is also an option. In this case, v0.berlin-gtfs-rt.transport.rest already contains the appropriate realtime data (with IDs matching the static/plan data, that's the point of the GTFS-Realtime format).

This already seems ideal. About the shapes I will have to see because I also saw https://github.com/derhuerst/vbb-shapes and I am not currently sure how it is connected.

vbb-shapes is built from the official static/plan data, but I think it's a bit outdated.

[creativecreatorormaybenot]

@derhuerst After thinking about it, the using the static shapes might be the less fragile and more efficient way of doing things, but if I understand the shapes API correctly, it should be way easier for me to poll each shape for every trip.

Now, I am just slightly worried it will get very messy. But I will see about that.

---

Thanks a lot for the estimation insights! That is super useful. Since I am only doing trains for now, I hope that they will be accurate enough to be able to work with them. But I think the only way to really test that is actually going outside and trying it :)

I will stick with Berlin for now - not because that is the ultimate goal, but because it seems like the only reasonable choice from a project management perspective ;)
Of course, it would be cool if eventually every city in the world is covered. But focusing on just Berlin should make this a lot more manageable!

[derhuerst]

[...] using the static shapes might be the less fragile and more efficient way of doing things, but if I understand the shapes API correctly, it should be way easier for me to poll each shape for every trip.

Which shapes API are you referring to here? The v5.vbb.transport.rest/shapes/{id} one? Or vbb-shapes?

Since both of these use the static/plan (GTFS) data's shape IDs anyways, and since I likely didn't update both to the latest static/plan (GTFS) dataset, it's probably best for you right now to use extract-gtfs-shapes with the latest GTFS dataset.

Sorry for the confusion, there are just too many half-maintained projects. 😅

[creativecreatorormaybenot]

@derhuerst Okay - I see. So the shapes are not pulled live according to e.g. current construction sites but static and need to be updated manually?

[derhuerst]

v5.vbb.transport.rest/shapes/{id} just exposes vbb-shapes via HTTP, and vbb-shapes would need to be updated whenever a new GTFS dataset has been published, which I haven't done in the past months. This is very confusing and should be cleaned up. If you use extract-gtfs-shapes with the latest VBB GTFS dataset by yourself, you can be sure to have up-to-date shapes.

All of these work with static shapes so they, while being trip-specific, only take planned construction work into account. If you want "realtime shapes", use the shapes returned by v5.vbb.trannsport.rest/trips/{id}?polyline=true.

[derhuerst]

Just stumbled upon another project that visualises GTFS Static data (and claims to ingest GTFS Realtime too) on a live map: https://github.com/vasile/transit-map

[creativecreatorormaybenot]

@derhuerst That looks awesome! Do they use the same data structure (given that it is not for Germany)?

[derhuerst]

Do they use the same data structure (given that it is not for Germany)?

There are GTFS (-Static) datasets for Berlin, Germany & Switzerland; All 3 datasets have the same structure & semantics, given that they follow the same spec (they might have a different level of details, and make use of different parts of GTFS, but the shapes part should be compatible).

So, yes, if you build a GTFS-based (optionally GTFS-Realtime support) solution, it will work for all regions where there's a GTFS (-Static) feed (and optionally a GTFS-Realtime feed) available.