This repository contains:
- Dockerfile(.template) of airflow for Docker images published to the public Docker Hub Registry.
- airflow.all.yaml for creating Kubernetes services and deployments to run Airflow on Kubernetes
- Highly inspired by the great work puckel/docker-airflow
- Based on Debian Jessie official Image debian:jessie and uses the official Postgres as backend and RabbitMQ as queue
- Following the Airflow release from Python Package Index
Create all the deployments and services for Airflow:
kubectl create -f airflow.all.yaml
git clone this repository and then just run:
make build
Create all the deployments and services to run Airflow on Kubernetse:
kubectl create -f airflow.all.yaml
It will create deployments for:
- postgres
- rabbitmq
- airflow-webserver
- airflow-scheduler
- airflow-flower
- airflow-worker
and services for:
- postgres
- rabbitmq
- airflow-webserver
- airflow-flower
You can browse the Airflow dashboard via running:
make browse-web
the Flower dashboard via running:
make browse-flower
If you want to use Ad hoc query, make sure you've configured connections:
Go to Admin -> Connections and Edit "mysql_default" set this values (equivalent to values in config/airflow.cfg) :
- Host : mysql
- Schema : airflow
- Login : airflow
- Password : airflow
Check Airflow Documentation
kubectl exec web-<id> --namespace airflow-dev airflow backfill tutorial -s 2015-05-01 -e 2015-06-01
For now, update the value for the replicas field of the deployment you want to scale and then:
make apply
Fork, improve and PR. ;-)