This repository is my vision of a "perfect" development environment for Airflow 2.X.
-
Development environment that closely mimics a production one:
- PostgreSQL is used as Metadata DB (contains information about the status of tasks, DAGs, Variables, connections, etc.) and queue results backend.
- Celery (with Flower) is a task queue.
- Redis is a message broker.
-
Dockerfile
anddocker-compose
files that are meant to be fully customizable through environment variables or.env
file. -
Development environment that enforces code quality and safety:
pytest
(andpytest-cov
) is used for unit- and integration testing.- Black enforces a consistent code format.
isort
takes care of imports sorting.- Flake8 (and
wemake-python-styleguide
) controls code style. - Bandit looks for vulnerability in code.
- Safety looks for insecure dependencies.