SMART MOBILITY SERVICE PLATFORM

vbb-rest-2-ngsi

Node.js application that uses the vbb-rest API to read public transport data of the Verkehrsverbund Berlin-Brandenburg (VBB) and transform it from FTPF into a NGSI-compliant data format (NGSI v2 and NGSI-LD) for subsequent storage in FIWARE NGSI Context Brokers. Applied FIWARE data models implement the General Transit Feed Specification (GTFS).

Current limitations

The NGSI data model types GtfsService and GtfsTrip are not considered yet. Although with regard to the relationship model these types should be harmonised, it's due to the implementation of the REST API that some information is not available or suitable for transformation into context data.
- A GtfsService should reference a GtfsAgency but various defined service times for routes are not known or cannot be queried via API.
- A GtfsTrip may reference a GtfsShape, should reference a GtfsService and a GtfsRoute, but a trip ID always includes date of the day and is often regenerated. Therefore its form is not suitable for transformation into a NGSI entity with fixed ID.
Due to runtime reasons, instead of querying all stations (~ 13000) from the vbb-rest API only a static subset of 10 selected stations is processed for now.
Furthermore just one possible journey of 10 selected connections (station A -> station B) is picked. It is planned to include all stops/stations and journeys at some point.

Prerequisites

Docker and Docker Compose must be installed on your system in order to run the services defined in the multi-container file.

Operation modes

The project offers two different compose files. The first variant starts the Node.js script, the context broker and components for persisting data. In this mode (client-server mode), the retrieved public transport data is stored in the context broker of the local Docker container and, if configured, persisted in the local CrateDB instance.

The second variant comprises a single service for the Node.js script. It acts as a client to a context broker running elsewhere (client mode).

Configuration

There is a configuration file .env containing environment variables used by the Node.js script. Some of the variables values have to be modified prior to initial startup, as the script uses those variables for connection management and data processing.

The following list gives a summary of currently supported variables and their description:

Name	Description	Default value
Application settings
`APP_START_DELAY`	delayed start: give other required services some time to get started before trying to connect to them [seconds] mandatory	20
VBB REST Data Source
`VBBREST_BASE_URL`	base URL (VBB REST data source) mandatory	https://3.vbb.transport.rest
`VBBREST_QUERY_INTERVAL`	interval for re-read data from REST API [seconds] NOTE: 24 hours (86400 sec.) as default, please do not stress this public API too much! mandatory	86400
NGSI v2 Context Broker
`BROKER_V2_BASE_URL`	NGSI v2 context broker URL NOTE: modify ONLY, when communicating with external context brokers, e.g. when executing `'./services-app-only'` which does NOT start any context broker! If this broker should not be used, just set an empty value. optional	http://orion-v2:1026/v2
`BROKER_V2_AUTH_KEY`	Auth key for 'Authorization' header optional
`BROKER_V2_API_KEY`	API key (token for authenticaton) optional
`BROKER_V2_TENANT`	tenant name (a tenant is a service aka domain on the context broker with its own isolated logical database) optional	vbb
`BROKER_V2_SUBTENANT`	sub-tenant name (a sub-tenant is a sub-service / service path aka project for the given tenant) optional	/public_transport
`BROKER_V2_ENTITY_ID_SUFFIX`	entity ID suffix (on creation will be appended to an entitys ID for a customized identification format, e.g. the ID suffix 'XY' for a `GtfsStop` entity '650030877901' will result in `GtfsStop:650030877901:XY`) optional
NGSI-LD Context Broker
`BROKER_LD_BASE_URL`	NGSI-LD context broker URL NOTE: modify ONLY, when communicating with external context brokers, e.g. when executing `'./services-app-only'` which does NOT start any context broker! If this broker should not be used, just set an empty value. optional	http://orion-ld:1026/ngsi-ld/v1
`BROKER_LD_AUTH_KEY`	Auth key for 'Authorization' header optional
`BROKER_LD_API_KEY`	API key (token for authenticaton) optional
`BROKER_LD_TENANT`	tenant name (a tenant is a service aka domain on the context broker with its own isolated logical database) optional	vbb
`BROKER_LD_SUBTENANT`	sub-tenant name (a sub-tenant is a sub-service / service path aka project for the given tenant) optional	/public_transport
`BROKER_LD_ENTITY_ID_SUFFIX`	entity ID suffix (on creation will be appended to an entitys ID for a customized identification format, e.g. the ID suffix 'XY' for a `GtfsStop` entity '650030877901' will result in `urn:ngsi-ld:GtfsStop:650030877901:XY`) optional
Historic data persistence
`ENABLE_HISTORIC_DATA_STORAGE`	enables storage of historic data (into Crate-DB via QuantumLeap API for now) - support for NGSI v2 data only optional	true
`QL_V2_NOTIFICATION_BASE_URL`	QuantumLeap (QL) notification URL used for sending status changes of entities in the context broker NOTE: modify ONLY, when communicating with external QL instances, e.g. when executing `'./services-app-only'` which does NOT start any QL instance! If historic data persistence via QL is not wanted, just set an empty value. optional	http://quantumleap:8668/v2
`QL_V2_AUTH_KEY`	Auth key for 'Authorization' header in requests to QL optional
`QL_V2_API_KEY`	API key for authentication on QL optional
`QL_V2_TENANT`	tenant name on QL optional	vbb
`QL_V2_SUBTENANT`	sub-tenant name on QL optional	/public_transport

Starting Docker containers

Depending on what operation mode is preferred, pull/create the images and start containers by running ./services create && ./services start (client-server mode) or simply ./services-app-only start (client mode) from the project root folder.
To stop the containers run ./services[-app-only] stop.
If you encounter problems executing the service scripts, add the missing permission with chmod +x services*.

GTFS entity relationships

NGSI data models

Read data from the vbb-rest API is harmonised for transformation into applicable FIWARE NGSI data models of the domain Urban Mobility.
Entites of the following types are sent to the context brokers:

GtfsStop - A stop where vehicles pick up or drop off riders
GtfsStation - A physical structure or area that contains one or more stops
GtfsAgency - Transit agency (transportation operator) that offers services at particular times
GtfsRoute - A (transit) route is a group of trips (sequences of two or more stops that occur during a specific time period) that are displayed to riders as a single service.
GtfsShape - Describes the path that a vehicle travels along a route alignment (consists of a sequence of points)

Entity IDs use the following format:

Entity Type	Entity ID format
`GtfsStop`	`GtfsStop:<STOP_ID>[:<ENTITY_ID_SUFFIX>]` e.g. GtfsStop:000008012713:XY
`GtfsStation`	`GtfsStation:<STATION_ID>[:<ENTITY_ID_SUFFIX>]` e.g. GtfsStation:900000245025:XY
`GtfsAgency`	`GtfsAgency:<AGENCY_ID>[:<ENTITY_ID_SUFFIX>]` e.g. GtfsAgency:berliner-verkehrsbetriebe:XY
`GtfsRoute`	`GtfsRoute:<ORIGIN_STATION_ID>-<DESTINATION_STATION_ID>[:<ENTITY_ID_SUFFIX>]` e.g. GtfsRoute:900000044201-900000009202:XY
`GtfsShape`	`GtfsShape:<ORIGIN_STATION_ID>-<DESTINATION_STATION_ID>[:<ENTITY_ID_SUFFIX>]` e.g. GtfsShape:900000044201-900000009202:XY

Notes:

<ENTITY_ID_SUFFIX> is an optional ID suffix that will be appended to each entity ID (preceded by a colon ':') if configured in the configuration.
The same rules apply to NGSI-LD entities. The only difference: urn:ngsi-ld: is prepended to the entity ID, e.g.
urn:ngsi-ld:GtfsStop:000008012713:XY`

Reading data from context brokers

You can GET a list of all entities using the following cURL commands. Don't forget to replace the <DOCKER_HOST> placeholders with the hostname / IP of your Docker host.
<DOCKER_HOST> assumes that you are running your own context brokers on the Docker host. If you are connecting to context brokers located elsewhere, use their hostname / IP address accordingly.

Orion v2

List all GtfsStation entities containing only the 'id' attribute:

curl -X GET '<DOCKER_HOST>:1026/v2/entities?type=GtfsStation&attrs=id&options=keyValues' \
  -H 'Accept: application/json'

List all GtfsStation entities containing all attributes:

curl -X GET '<DOCKER_HOST>:1026/v2/entities?type=GtfsStation&options=keyValues' \
  -H 'Accept: application/json'

History data

If historic data persistence was enabled, the Node.js script sends subscriptions for attribute changes of all known entities to the NGSI v2 context broker. QuantumLeap, a REST service for storing, querying and retrieving spatial-temporal data, will receive a notification every time a status changes and stores its current value in a CrateDB database. With this data collected over time, statistical evaluations and data visualisation will be possible, e.g. building histograms with Grafana or UI widgets using WireCloud.

As QuantumLeap has no support for NGSI-LD yet, storage of historic data is supported for NGSI v2 data only.

The Docker container of CrateDB exposes 4200 as the default port for data queries and access to the web-based admin UI.
You can reach it at <DOCKER_HOST>:4200.

Troubleshooting

CrateDB service might crash shortly after startup due to incompatible memory settings.
With docker ps -a you can check whether its container is running or has already exited. In the latter case, inspecting the container log file with

sudo vi `docker inspect --format='{{.LogPath}}' kipark-db-crate`

should give you an output saying something like
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144].

In order to avoid this, increase maximum number of memory map areas before starting:

sudo sysctl -w vm.max_map_count=262144

Known issues

Due to some open issues of the NGSI-LD context broker, unexpected behaviour and errors may appear while executing the script.

Preceding incomplete query results, causing the Node.js script to attempt to recreate existing entities:
FIWARE/context.Orion-LD#405
Sudden interruption of socket connections to the context broker, making the Orion-LD unreachable afterwards:
FIWARE/context.Orion-LD#408
Unmotivated crashes of the Orion-LD, which also makes the broker unreachable:
FIWARE/context.Orion-LD#406

License

This project is licensed under MIT License.

SmartMaaS-Services/vbb-rest-2-ngsi