Kedro Airflow K8S Plugin
About
The main purpose of this plugin is to enable running kedro pipeline with Airflow on Kubernetes Cluster. In difference to kedro-airflow this plugin does not require additional libraries installed in airflow runtime, it uses K8S infrastructure instead. It supports translation from Kedro pipeline DSL to airflow (python API) and generation of airflow DAGs.
The plugin can be used together with kedro-docker to simplify preparation of docker image for pipeline execution.
Documentation
For detailed documentation refer to https://kedro-airflow-k8s.readthedocs.io/
Usage guide
Usage: kedro airflow-k8s [OPTIONS] COMMAND [ARGS]...
Options:
-e, --env TEXT Environment to use.
-p, --pipeline TEXT Pipeline name to pick.
-h, --help Show this message and exit.
Commands:
compile Create an Airflow DAG for a project
init Initializes configuration for the plugin
list-pipelines List pipelines generated by this plugin
run-once Uploads pipeline to Airflow and runs once
schedule Uploads pipeline to Airflow with given schedule
ui Open Apache Airflow UI in new browser tab
upload-pipeline Uploads pipeline to Airflow DAG location