eandualem/sensor_data_elt

Sensor Data ELT

A fully dockerized ELT pipeline using MySQL, PostgreSQL, Airflow, DBT, Redash and Superset.
Explore the docs »

· Report Bug · Request Feature .

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgements

About The Project

The objective of this project was to migrate an ELT pipeline developed for the week 11 challenge using(MYSQL, DBT, Apache Airflow, and Redash) to a more scalable and robust ELT pipeline. This was accomplished by changing the two main components, namely the MySQL data warehouse to Postgres and the Redash dashboard to Superset.

Built With

Tech Stack used in this project

• MYSQL
• PostgreSQL
• Apache Airflow
• dbt
• Redash
• Superset

Getting Started

Prerequisites

Make sure you have docker installed on local machine.

• Docker
• DockerCompose

Installation

1. Clone the repo

   git clone https://github.com/eandualem/sensor_data_elt

2. Datawarehouse

   cd sensor_data_elt

3. Run

    docker-compose up

4. Can access and Modefy the default configrations for each tool using the .env files.

Usage

Adminer:

Adminer (formerly phpMinAdmin) is a full-featured database management tool written in PHP. Used to access MYSQL and Postgres Databases.

• MYSQL:

  Navigate to `http://localhost:8080/` on the browser
  use `mysqldb` server
  use `dbtdb` database
  use `root` for username
  use `root` for password

• Postgress:

  Navigate to `http://localhost:8080/` on the browser
  use `postgres-dbt` server
  use `dbtdb` database
  use `dbtuser` for username
  use `pssd` for password

Airflow:

Airflow is used for aurchestration and automation.

Navigate to `http://localhost:8080/` on the browser
use `admin` for username
use `admin` for password

DBT:

DBT is used for cleaning and transforming the data in the warehouses.

• Airflow is used for automation of running and testing dbt models
• navigate to https://sensordataelt.herokuapp.com/index.html to access dbt docs

Redash

open terminal and execute `docker-compose run — rm server create_db`
using adminer create a user and grant read access
Navigate to `http://localhost:5000/` on the browser

Superset

• navigate to localhost:8088 to access Airflow

Roadmap

See the open issues for a list of proposed features (and known issues).

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Elias Andualem - eandualem@gmail.com

Project Link: https://github.com/eandualem/sensor_data_elt

Acknowledgements

• 10 Academy