Registry DAG Template
Astronomer Registry.
Maintained with ❤️ by Astronomer.
Guidelines on building, organizing, and publishing example DAGs for the Airflow community to discover and find inspiration from on the For the best user experience by the Airflow community, example DAG repositories should be public on GitHub and follow the structural guidelines and practices presented below.
Before you begin
An easy way to get started developing and running Apache Airflow pipelines locally is with the Astro CLI. Learn how to install it here. The CLI is also a great way to take advantage of this repository template.
Repository structure
The organization of this repository resembles that generated by a project initialized by the Astro CLI. This template provides all the necessary files to run Apache Airflow locally within Docker containers using the Astro CLI along with an example DAG.
.
├── .astro
│ └── config.yaml # Configuration file for local environment settings
├── .dockerignore
├── .gitignore
├── Dockerfile
├── README.md
├── dags # DAG files go here
│ └── example_dag.py
├── packages.txt # For OS-level packages
├── requirements.txt # Place Provider packages as well as other Python modules used in your Airflow DAG here
└── .astro-registry.yaml # Contains the path.to.dag_file of DAGs to publish to the Astronomer Registry as well as the categories they should be assigned to.
Here are additional details about these files plus other directories and files that are optional to include in your own repository:
.astro/config.yaml
This file is used to establish settings to your local environment. The project.name
setting is declared in the file already to "registry_dag_template". Ideally this should be updated to a pertinent name for the project however.
Dockerfile
The Dockerfile includes a reference to an Astronomer Certified Docker Image. Astronomer Certified (AC) is a Debian-based, production-ready distribution of Apache Airflow that mirrors the open source project and undergoes additional levels of rigorous testing conducted by the Astronomer team.
This Docker image is hosted on Astronomer's Docker Registry and allows you to run Airflow on Astronomer. Additionally, the image you include in your Dockerfile dictates the version of Airflow you'd like to run locally.
Note: Airflow version 1.10.x reached end-of-life on June 17, 2021. It is strongly recommended to use a version of Airflow that is 2.0 or greater.
README.md
The README for your example DAG repository has an understated importance -- it provides necessary context for the Airflow community about your DAG. Feel free to include a use case description or a reason why the DAG was created or even how it can be used, the Airflow Providers used, the connections that are assumed to be in place to run the DAG, etc. The more details, the better.
requirements.txt
Any additional Python modules that are necessary to install in your local Airflow environment should be listed here. This includes Provider packages which are used in the DAG. For transparency of what versions of modules were used during development and testing of the DAG, the versions should be pinned. This allows others to easily run, reproduce, and build upon the DAG while not having to guess what versions of modules were installed.
For example:
apache-airflow-providers-amazon==2.0.0
apache-airflow-providers-salesforce==3.0.0
apache-airflow-providers-snowflake==2.0.0
jsonschema==3.2.0
Browse the Astronomer Registry to find all of the available Providers that can be used in any Airflow DAG.
.astro-registry.yaml
This file is used to help control what DAGs should be published on the Astronomer Registry and what categor(y/ies) should be assigned to the DAGs in repository on the Registry. This file should be kept up-to-date with any new DAGs that should be published to the Registry. (More info in the Publishing your DAG repository for the Astronomer Registry section.)
The format is straightforward:
# These categories will be applied to all DAGs in the repo.
categories:
- Airflow Fundamentals
# List of DAGs that should be published to the Astronomer Registry.
dags:
- path: dags/example_dags.py # Must be the path.to.the.dag_file.py
More that one category can be assigned to the DAGs. The following DAG categories are currently supported on the Astronomer Registry so you pick the ones the best described your repo of DAGs:
- AI + Machine Learning
- Airflow Fundamentals
- Alerts/Notifications
- Big Data & Analytics
- CI/CD
- Communication & Messaging
- Compute
- Containers
- Data Management & Governance
- Data Processing
- Data Quality
- Data Science
- Data Storage
- Databases
- DevOps
- ETL/ELT
- Infrastructure (IaaS)
- Logging & Monitoring
- Machine Learning
- Orchestration
- Public Cloud
- Query Engines
- Web Services
- Work Management
include/
[Optional]
This directory can be added as a main directory to the repository (i.e. same level as dags/
) to house any other files. This can be Python functions, small/reference datasets, custom Airflow Operators
, SQL files, etc. With the Astro CLI, this directory is automatically read from so changes to any files in this directory will be picked up without having to restart your local environment.
As part of a DAG-writing best practice, it is a good idea to separate accompanying logic and data from the logic which creates the DAG
. Think of the DAG file as a configuration file; it should be clean and ideally only contain logic for the DAG
object, tasks, and task dependencies. Using the include/
directory can help improve organization of the repository greatly by providing a place to put the other "stuff".
For example:
...
├── include
│ ├── operators
│ │ ├── __init__.py
│ │ └── custom_operator.py
│ └── sql
│ │ ├── extract
│ │ │ └── extract_data.sql
│ │ ├── load
│ │ │ └── insert_into_table.sql
│ │ └── transform
│ │ └── transform_other_data.sql
│ └── data
│ └── reference_data.csv
...
Before creating custom
Operators
or using thePythonOperator
to execute logic, check out the Astronomer Registry to find all of the available Modules that can be used out-of-the-box with Airflow Providers.
plugins/
[Optional]
This directory can be added as a main directory to the repository (i.e. same level as dags/
) to house any logic for Airflow UI plugins like menu links. With the Astro CLI, this directory is automatically read from so changes to any files in this directory will be picked up without having to restart your local environment.
airflow_settings.yaml
[Optional]
When using the Astro CLI, this file can be used to programmatically create Connections, Variables, and Pools for your local Airflow environment. For more information about this file, check out this documentation.
Note: Since this file can contain sensitive information like credentials, it is strongly recommended that this file be used for local development only and not be published to GitHub. The
.gitignore
within this repository template file does list theairflow_settings.yaml
file so please do not remove the entry.
Using this template
Follow the steps below to use the template and initialize a new repository locally.
-
Above the file list, click Use this template
-
Type a name for the repository, select your user as the owner, and an optional description.
-
Select Public visibility for the repository.
-
Click Create repository from template.
-
Clone the repository locally. Refer to the GitHub documentation for different options of cloning repositories.
-
[Optional] Navigate to where the repository was cloned and run the following Astro CLI command to update the
project.name
setting in the.astro/config.yaml
file provided in the repository. This will update the name used to generate Docker containers and make them more discernible if there are multiple DAGs initialized locally via this template.astro config set project.name <name-of-repository>
-
To beginning development and testing of the DAG, run
astro dev start
to spin up a local Airflow environment. There is no need to runastro dev init
as this functionality is already built in to the template repository.
Key requirements
-
The DAG must have a top-level docstring. The docstring should comprise of the title of DAG on a first, standalone line (this will be used as the title displayed on the Astronomer Registry), then a following paragraph describing the DAG itself. This will be used as the DAG description on the Registry.
-
The repo must have at least one semantically versioned tag. Repository tags are how the Astronomer Registry will know new DAG updates have occurred and to propagate those updates to the Registry. (More on this in the next section.)
Publishing your DAG repository for the Astronomer Registry
If you have never submitted your DAG repository for publication to the Astronomer Registry, create a new release/tag for your repository on the main
branch. Ultimately, the backend of the Astronomer Registry will check for new tags for a DAG repository to trigger updating the DAG on the Registry.
NOTE: Tags for the repository must follow typical semantic versioning.
Now that you've created a release/tag, head over to the Astronomer Registry and fill out the form with your shiny new DAG repo details!
If your repo has already been published to the Astronomer Registry, create a new release/tag and the Astronomer Registry will do the rest (picking up updates to DAG files, new DAGs, etc.). Just make sure the .astro-registry.yaml
file is updated with any new or updated DAGs.