/DE_demo

Primary LanguageHTML

How to run

This repo contains a runnable demo using Astronomer (containerized Airflow), which is a convenient option to run everything in a Docker container.

  • Install the Astronomer CLI (containerized Airflow), instructions here
    • Note: If you only want to run Airflow locally for development, you do not need to sign up for an Astronomer Cloud account. Simply follow the instructions to install the Astronomer CLI.
  • Run astro dev start to start up the Airflow Docker containers
    • I had to follow the Docker config instructions here to handle a "buildkit not supported" error
    • I also had to reduce the number of AIRFLOW__WEBSERVER__WORKERS in the Dockerfile as well as allocate more resources to Docker in order for the webserver to run on my very old very slow laptop :) (2013 MacBook Air ftw)
    • Thanks to this post for the agate version pin to work with dbt
  • This will start up the Airflow scheduler, webserver, and a Postgres database
  • Once the webserver is up (takes about 30 seconds), you can access the Airflow web UI at localhost:8080
  • You can run astro dev stop to stop the container again

You can also run the DAG in this repo with a standard Airflow installation if you want. You'll have to install the relevant dependencies (Airflow, dbt, Great Expectations, the respective operators, etc) and probably handle some more configurations to get it to work.