A plugin for Apache Airflow that allows you to edit DAGs in browser. It provides a file managing interface within specified directories and it can be used to edit, upload, and download your files. If git support is enabled, the DAGs are stored in a Git repository. You may use it to view Git history, review local changes and commit.
- Airflow Versions
- 1.10.3 or newer
- git Versions (git is not required if git support is disabled)
- 2.0 or newer
For the ease of deployment, use the production-ready reference container image. The image is based on the reference images for Apache Airflow.
You can find the following images there:
- andreax79/airflow-code-editor:latest - the latest released Airflow Code Editor image with the latest Apache Airflow version
- andreax79/airflow-code-editor:2.7.0 - the latest released Airflow Code Editor with specific Airflow version
- andreax79/airflow-code-editor:2.7.0-7.4.1 - specific version of Airflow and Airflow Code Editor
- Install the plugin
pip install airflow-code-editor
- Install optional dependencies
-
black - Black Python code formatter
-
fs-s3fs - S3FS Amazon S3 Filesystem
-
fs-gcsfs - Google Cloud Storage Filesystem
-
... other filesystems supported by PyFilesystem - see https://www.pyfilesystem.org/page/index-of-filesystems/
pip install black fs-s3fs fs-gcsfs
-
Restart the Airflow Web Server
-
Open Admin - DAGs Code Editor
You can set options editing the Airflow's configuration file or setting environment variables. You can edit your airflow.cfg adding any of the following settings in the [code_editor] section. All the settings are optional.
- enabled enable this plugin (default: True).
- git_enabled enable git support (default: True). If git is not installed, disable this option.
- git_cmd git command (path)
- git_default_args git arguments added to each call (default: -c color.ui=true)
- git_author_name human-readable name in the author/committer (default logged user first and last names)
- git_author_email email for the author/committer (default: logged user email)
- git_init_repo initialize a git repo in DAGs folder (default: True)
- root_directory root folder (default: Airflow DAGs folder)
- line_length Python code formatter - max line length (default: 88)
- string_normalization Python code formatter - if true normalize string quotes and prefixes (default: False)
- mount, mount1, ... configure additional folder (mount point) - format: name=xxx,path=yyy
- ignored_entries comma-separated list of entries to be excluded from file/directory list (default: .*,__pycache__)
[code_editor]
enabled = True
git_enabled = True
git_cmd = /usr/bin/git
git_default_args = -c color.ui=true
git_init_repo = False
root_directory = /home/airflow/dags
line_length = 88
string_normalization = False
mount = name=data,path=/home/airflow/data
mount1 = name=logs,path=/home/airflow/logs
mount2 = name=data,path=s3://example
Mount Options:
- name: mount name (destination)
- path: local path or PyFilesystem FS URLs - see https://docs.pyfilesystem.org/en/latest/openers.html
Example:
- name=ftp_server,path=ftp://user:pass@ftp.example.org/private
- name=data,path=s3://example
- name=tmp,path=/tmp
You can also set options with the following environment variables:
- AIRFLOW__CODE_EDITOR__ENABLED
- AIRFLOW__CODE_EDITOR__GIT_ENABLED
- AIRFLOW__CODE_EDITOR__GIT_CMD
- AIRFLOW__CODE_EDITOR__GIT_DEFAULT_ARGS
- AIRFLOW__CODE_EDITOR__GIT_AUTHOR_NAME
- AIRFLOW__CODE_EDITOR__GIT_AUTHOR_EMAIL
- AIRFLOW__CODE_EDITOR__GIT_INIT_REPO
- AIRFLOW__CODE_EDITOR__ROOT_DIRECTORY
- AIRFLOW__CODE_EDITOR__LINE_LENGTH
- AIRFLOW__CODE_EDITOR__STRING_NORMALIZATION
- AIRFLOW__CODE_EDITOR__MOUNT, AIRFLOW__CODE_EDITOR__MOUNT1, AIRFLOW__CODE_EDITOR__MOUNT2, ...
- AIRFLOW__CODE_EDITOR__IGNORED_ENTRIES
Example:
export AIRFLOW__CODE_EDITOR__STRING_NORMALIZATION=True
export AIRFLOW__CODE_EDITOR__MOUNT='name=data,path=/home/airflow/data'
export AIRFLOW__CODE_EDITOR__MOUNT1='name=logs,path=/home/airflow/logs'
export AIRFLOW__CODE_EDITOR__MOUNT2='name=tmp,path=/tmp'
-
Fork the repo
-
Clone it on the local machine
git clone https://github.com/andreax79/airflow-code-editor.git
cd airflow-code-editor
- Create dev image
make dev-image
- Switch node version
nvm use
- Make changes you need. Build npm package with:
make npm-build
- You can start Airflow webserver with:
make webserver
- Run tests
make test
-
Commit and push changes
-
Create pull request to the original repo
- Apache Airflow
- Codemirror, In-browser code editor
- Git WebUI, A standalone local web based user interface for git repositories
- Black, The Uncompromising Code Formatter
- pss, power-tool for searching source files
- Vue.js
- Vue-good-table, data table for VueJS
- Vue-tree, TreeView control for VueJS
- Vue-universal-modal Universal modal plugin for Vue@3
- Vue-simple-context-menu
- Splitpanes
- Axios, Promise based HTTP client for the browser and node.js
- PyFilesystem2, Python's Filesystem abstraction layer
- Amazon S3 PyFilesystem
- Google Cloud Storage PyFilesystem