idarlington/nsdiruptions

Scala

NS-Disruptions

Scrapes the disruptions from api provided by the Dutch train service (NS)

Getting Started

This project is built using

Scala
FS2
Flink
HTTP4S
Kafka
Postgres

It contains a couple of microservices

Scraper service that scrapes disruptions from the NS API and dumps in a Kafka topic
Decoder service that decodes this data and dumps successfully decoded data into Kafka
DeDuplicator service that filters out duplicate entries and store single entries in a kafka topic
DatabaseSink

Running the project

Docker Compose

To run the project, a docker-compose file is provided.

From the base directory, you can run

sbt docker:publishLocal && docker-compose -f docker/docker-compose.yml up

Helm

Assuming you already have a Kubernetes cluster setup, you can start one locally with minikube. You can run the project on it.

You can start with exporting your NS API Keys

export SCRAPER_AUTH_KEY=

Run the helm installation script for the provided chart

Running the tests

sbt test