Demo pipeline for Airflow 2.x: fetches BTC/USD converstion rate from exchangerate.host and stores it in Postgres DB.
docker-compose.yaml based on recommended docker-compose from Airflow community. Also, postgres container added.
Steps to run:
- Start containers:
docker-compose up -d
- Create DB connections:
docker-compose run airflow-worker airflow connections add 'dwh_db' \
--conn-uri postgresql+psycopg2://airflow:airflow@postgres-dwh/dwh
- Run DAG
btc_usd_demo_etl
from UI (default login/pass:airflow/airflow
) or via CLI:
docker-compose run airflow-worker airflow dags backfill btc_usd_demo_etl --start-date $(date "+%Y-%m-%d")
- Check DAG results in postgres:
docker-compose exec postgres-dwh psql -U airflow dwh -c "select * from rates"