This is a collection of Airflow operators to provide easy integration with dbt.
Forked for: Add feature to allow environment variables to be passed to the dbt command line, so that profiles.yml can receive secrets at runtime from the airflow DAG. This improves management of airflow-dbt pipelines on AWS MWAA with secrets pulled from an aws secrets store.
from airflow import DAG
from airflow_dbt.operators.dbt_operator import (
DbtSeedOperator,
DbtSnapshotOperator,
DbtRunOperator,
DbtTestOperator
)
from airflow.utils.dates import days_ago
default_args = {
'dir': '/srv/app/dbt',
'start_date': days_ago(0)
}
with DAG(dag_id='dbt', default_args=default_args, schedule_interval='@daily') as dag:
dbt_seed = DbtSeedOperator(
task_id='dbt_seed',
)
dbt_snapshot = DbtSnapshotOperator(
task_id='dbt_snapshot',
)
dbt_run = DbtRunOperator(
task_id='dbt_run',
)
dbt_test = DbtTestOperator(
task_id='dbt_test',
retries=0, # Failing tests would fail the task, and we don't want Airflow to try again
)
dbt_seed >> dbt_snapshot >> dbt_run >> dbt_test
Install from PyPI:
pip install airflow-dbt
It will also need access to the dbt
CLI, which should either be on your PATH
or can be set with the dbt_bin
argument in each operator.
There are four operators currently implemented:
DbtSeedOperator
- Calls
dbt seed
- Calls
DbtSnapshotOperator
- Calls
dbt snapshot
- Calls
DbtRunOperator
- Calls
dbt run
- Calls
DbtTestOperator
- Calls
dbt test
- Calls
Each of the above operators accept the following arguments:
profiles_dir
- If set, passed as the
--profiles-dir
argument to thedbt
command
- If set, passed as the
target
- If set, passed as the
--target
argument to thedbt
command
- If set, passed as the
dir
- The directory to run the
dbt
command in
- The directory to run the
full_refresh
- If set to
True
, passes--full-refresh
- If set to
vars
- If set, passed as the
--vars
argument to thedbt
command. Should be set as a Python dictionary, as will be passed to thedbt
command as YAML
- If set, passed as the
environ
- If set, values in the passed Python dictionary will be added as supplemental environment variables to the environment that the
dbt
command runs within, enabling secrets to be passed to profiles.yml at runtime.
- If set, values in the passed Python dictionary will be added as supplemental environment variables to the environment that the
models
- If set, passed as the
--models
argument to thedbt
command
- If set, passed as the
exclude
- If set, passed as the
--exclude
argument to thedbt
command
- If set, passed as the
select
- If set, passed as the
--select
argument to thedbt
command
- If set, passed as the
dbt_bin
- The
dbt
CLI. Defaults todbt
, so assumes it's on yourPATH
- The
verbose
- The operator will log verbosely to the Airflow logs
Typically you will want to use the DbtRunOperator
, followed by the DbtTestOperator
, as shown earlier.
You can also use the hook directly. Typically this can be used for when you need to combine the dbt
command with another task in the same operators, for example running dbt docs
and uploading the docs to somewhere they can be served from.
To install from the repository: First it's recommended to create a virtual environment:
python3 -m venv .venv
source .venv/bin/activate
Install using pip
:
pip install .
To run tests locally, first create a virtual environment (see Building Locally section)
Install dependencies:
pip install . pytest
Run the tests:
pytest tests/
This project uses flake8.
To check your code, first create a virtual environment (see Building Locally section):
pip install flake8
flake8 airflow_dbt/ tests/ setup.py
If you use dbt's package manager you should include all dependencies before deploying your dbt project.
For Docker users, packages specified in packages.yml
should be included as part your docker image by calling dbt deps
in your Dockerfile
.
- This is available as open source under the terms of the MIT License.
- Bug reports and pull requests are welcome on GitHub at https://github.com/gocardless/airflow-dbt.
GoCardless ♥ open source. If you do too, come join us.