This project aims to monitor and analyze traffic data in real-time and batch processing mode. The architecture is based on Apache Spark, HDFS, Spark Streaming, Apache Kafka, InfluxDB, PostgreSQL, Metabase. The project is divided into three main components:
- Batch Processing: This component is responsible for ETL processes, loading data from HDFS, transforming it, and saving it to PostgreSQL. It also includes SQL queries used for creating dashboards and visualizations with Metabase.
- Environment: This component provides Docker Compose and other configuration files necessary for setting up the infrastructure for the project.
- Streaming: This component implements a Kafka producer to simulate car traffic data (latitude, longitude, and speed). The Spark Streaming processes this data and writes it to InfluxDB and PostgreSQL.
To set up the project, follow these steps:
- Clone the repository:
git clone - Change to the project directory:
cd traffic-monitoring - Set up the environment using Docker Compose:
cd environment; docker-compose up -d - Follow the instructions in each component's README file to run and test the application.
batch_processing: Contains ETL process, HDFS loading, data transformation, and saving to PostgreSQL. It also includes the SQL queries used for creating dashboards with Metabase.environment: Includes the Docker Compose file and other configuration files necessary for setting up the infrastructure for the project.streaming: Contains the Kafka producer and Spark Streaming code to process the data coming from simulated car traffic.
For more details, please refer to each folder's README file.