Traffic-data-ELT-pipeline

An ELT pipeline built for the pNEUMA open dataset of naturalistic trajectories of half a million vehicles collected by a swarm of drones in a congested downtown area of Athens, Greece.

Data Warehousing

A dockerized ELT pipeline.

Explore the docs »
Report Bug · Request Feature

Table of Contents

About The Project
- Built With
Getting Started
- Prerequisites
- Installation
Contributing
License
Contact
Acknowledgements

About The Project

A dockerized Extract, Load, Transform (ELT) pipeline with PostgreSQL, Airflow, DBT, and a Redash.

Built With

Tech Stack used in this project includes:

Getting Started

Prerequisites

Make sure you have docker installed on local machine.

Docker
Docker Compose

Installation

Clone the repo

git clone https://https://github.com/teddyk251/traffic-data-ELT-pipeline.git

Navigate to the folder
```
cd traffic-data-ELT-pipeline/airflow
```

Build an airflow image

docker build . --tag apache_dbt/airflow:2.3.3

Run
```
 docker-compose up
```

Open Airflow web browser

Navigate to `http://localhost:8080/` on the browser
activate and trigger load_dag
activate and trigger dbt_dag

Access redash dashboard

Navigate to `http://localhost:5000/` on the browser

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

Fork the Project
Create your Feature Branch (git checkout -b feature/AmazingFeature)
Commit your Changes (git commit -m 'Add some AmazingFeature')
Push to the Branch (git push origin feature/AmazingFeature)
Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Acknowledgments

10 Academy