A scalable data migration from a pipeline initially built using Postgres as the data warehouse to MySQL. The migration also includes a change of the visualization dashboard from Redash to Apache Superset.
A dockerized ELT pipeline.
Explore the docs »
Report Bug
·
Request Feature
Table of Contents
A dockerized Extract, Load, Transform (ELT) pipeline with MySQL, Airflow, DBT, and Apache Superset.
Tech Stack used in this project includes:
Make sure you have docker installed on local machine.
- Docker
- Docker Compose
- Clone the repo
git clone https://https://github.com/teddyk251/traffic-data-ELT-pipeline.git
- Navigate to the folder
cd traffic-data-ELT-pipeline/airflow
- Build an airflow image
docker build . --tag apache_dbt/airflow:2.3.3
- Run
docker-compose up
- Open Airflow web browser
Navigate to `http://localhost:8080/` on the browser activate and trigger load_dag activate and trigger dbt_dag
- Access Superset dashboard
Navigate to `http://localhost:8088/` on the browser
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature
) - Commit your Changes (
git commit -m 'Add some AmazingFeature'
) - Push to the Branch (
git push origin feature/AmazingFeature
) - Open a Pull Request
Distributed under the MIT License. See LICENSE
for more information.