Pre-requisites
- Install Docker https://docs.docker.com/engine/install/
How to run the project (on Unix based OS)
- run
airflow_up.sh
- wait until all docker containers are up and running
- run
add_airflow_connection.sh
to add postgres connection to airflow - check docker container health by executing
docker ps
- If
airflow-webserver
is still unhealthy, wait and repeat from step 4- the wait time is usually around 7-15 minutes
- visit
localhost:8080
(the username and password areadmin
andadmin
respectively) to open Airflow Webserver UI - turn-on
raw_to_clean_daily_12.00_PM
andraw_to_clean_weekly_sunday_12.00_PM
DAGs- If you get an error, the most likely cause is the database and schema is not defined properly. If airflow logs confirms that assumption, you can execute
seed_postgres_ecommerce.sh
and clear tasks (this should fix the issue)
- If you get an error, the most likely cause is the database and schema is not defined properly. If airflow logs confirms that assumption, you can execute
- wait until
raw_to_clean_daily_12.00_PM
DAG has ran successfully 791 times (this will can take several hours) - trigger
marketing_RFM_views
DAG manually - run
airflow_down.sh
to stop airflow containers. - run
superset_up.sh
to turn on Apache Superset container. - visit
localhost:8088
(the username and password areadmin
andadmin
respectively) to open Apache Superset. [Marketing] Recency, Frequency, Monetary Dashboard
dashboard should be available inDashboards
tab.- run
down.sh
when you're done.