Airflow-Boilerplate
- Airflow based DAG(directed acyclic graph) workflows management system.
Prerequisites
Please install direnv, pyenv, and pyenv-virtualenv for the local development.
.direnvrc
Please copy the .direnvrc file to your $HOME directory. It allows some bash functions declared in .direnvrc file to be available in .envrc.
cp .direnvrc $HOME
After having the installation the above packages, you can see the folllowing results:
$ git clone git@github.com:ziwon/airflow-boilerplate.git
$ cd airflow-boilerplate
direnv: loading .envrc
direnv: using python 3.5.5
direnv: export +AIRFLOW_API_AUTH +AIRFLOW_DATABASE_URL +AIRFLOW_EXECUTOR +AIRFLOW_HOME +AIRFLOW_RESULT_BACKEND +VIRTUAL_ENV ~PATH
Development
Both of Web UI and Scheduler should be launched separately in the local environment.
Launch the Web UI server:
$ cd airflow-boilerplate
direnv: loading .envrc
direnv: using python 3.5.5
direnv: export +AIRFLOW_API_AUTH +AIRFLOW_DATABASE_URL +AIRFLOW_EXECUTOR +AIRFLOW_HOME +AIRFLOW_RESULT_BACKEND +VIRTUAL_ENV ~PATH
$ ./bin/run web
Lunch the Scheduler in other console:
$ cd airflow-boilerplate
direnv: loading .envrc
direnv: using python 3.5.5
direnv: export +AIRFLOW_API_AUTH +AIRFLOW_DATABASE_URL +AIRFLOW_EXECUTOR +AIRFLOW_HOME +AIRFLOW_RESULT_BACKEND +VIRTUAL_ENV ~PATH
$ ./bin/run scheduler
Running
Please wait patiently, due to taking ages to build panas in alpine (While your computer is building this image, you might be able to cook your Thanksgiving turkey). Or you can change it to the prebuilt image to get done in short.
make build
make start