Setup Airflow Instance
Closed this issue · 1 comments
As a developer, I want to use Airflow to manage CRON jobs. Currently, we are using crontab to run periodic tasks and tracking execution via https://healthchecks.io
Airflow is a workhorse scheduler that can allow us to schedule periodic tasks with Python; we can track the results of our schedule jobs in an attractive UI. Can also use the UI to trigger jobs as needed.
The goal is to bring a self-hosted Airflow instance into the Busy Beaver project. This should be a repo inside of the busy-beaver-dev organization with a good story around CICD. We want to be able to deploy new DAGs by merging into master.
A few months back, I set up an Airflow installation in Docker-Compose This can be used as a template.
Using Kubernetes Cron Jobs to run periodic tasks.
This is out of scope.