Dagermeister 🤯
Build complex Airflow DAGs using drag-and-drop
Data engineering has never been easier.
Dagermeister is powered by Typescript, Pydantic, FastAPI and Airflow's Dagbag. It produces a Python DAG file that can be easily deployed to your Airflow cluster.
# -*- coding: utf-8 -*-
# This file was generated by a DAG factory engine
# Do not edit this file directly. Your changes will be overwritten by any working CI-CD pipeline.
from airflow import DAG
import dagfactory
# declare configuration
configuration = {'dag': {
'default_args': {
'owner': 'example_owner',
'start_date': '2023-01-01',
'end_date': '2023-01-05',
'retries': 1,
'retry_delay_sec': 300,
},
'schedule_interval': '0 3 * * *',
'concurrency': 1,
'max_active_runs': 1,
'dagrun_timeout_sec': 60,
'default_view': 'tree',
'orientation': 'LR',
'description': 'Generated by Airflow-UI',
'tasks': {
'2': {'operator': 'BashOperator', 'dependencies': ['1']},
'3': {'operator': 'DummyOperator', 'dependencies': ['1']},
'4': {
'operator': 'PythonOperator',
'dependencies': ['2'],
'task_id': '',
'python_callable': "def main():\n print('idk')",
'op_kwargs': '',
},
'test': {
'operator': 'PythonOperator',
'dependencies': ['4'],
'task_id': '',
'python_callable': "def main():\n print('hello world')",
'op_kwargs': '',
},
},
}}
dag_factory = dagfactory.DagFactory(config=configuration)
dag_factory.clean_dags(globals())
dag_factory.generate_dags(globals())The DagFactory itself is built on top of @ajbosco's dag-factory.
Important Notes
I have actively been designing the Dagermeister's architecture over the past year and there is still a lot to do. I am open-sourcing this at the moment for feature suggestions from the data engineering community.