teddyk251/traffic_data_warehouse_migration

Data Warehouse Migration from PostgreSQL to MySQL

Traffic-data-ELT-pipeline Data Migration

A scalable data migration from a pipeline initially built using Postgres as the data warehouse to MySQL. The migration also includes a change of the visualization dashboard from Redash to Apache Superset.

Data Warehousing

A dockerized ELT pipeline.

Explore the docs »
Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Contributing
4. License
5. Contact
6. Acknowledgements

About The Project

A dockerized Extract, Load, Transform (ELT) pipeline with MySQL, Airflow, DBT, and Apache Superset.

Built With

Tech Stack used in this project includes:

• 
• 
• 
• 
• 

Getting Started

Prerequisites

Make sure you have docker installed on local machine.

• Docker
• Docker Compose

Installation

1. Clone the repo

   git clone https://https://github.com/teddyk251/traffic-data-ELT-pipeline.git

2. Navigate to the folder

   cd traffic-data-ELT-pipeline/airflow

3. Build an airflow image

   docker build . --tag apache_dbt/airflow:2.3.3

4. Run

    docker-compose up

5. Open Airflow web browser

   Navigate to `http://localhost:8080/` on the browser
   activate and trigger load_dag
   activate and trigger dbt_dag

6. Access Superset dashboard

   Navigate to `http://localhost:8088/` on the browser

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Acknowledgments

• 10 Academy