Airflow in Kubernetes Executor

Pre-requisite

Download Docker Desktop from [here](https://www.docker.com/products/docker-desktop] and setup in your local machine)
Download Virtualbox from here and setup in your local machine

Install Minikube, Helm & add package repo

bash
brew update

# install minikube for local development
brew install minikube

# install helm
brew install kubernetes-helm

# add package repository to helm
helm repo add stable https://kubernetes-charts.storage.googleapis.com

# initialize docker-machine
docker-machine start
docker-machine env
eval $(docker-machine env)

Deployment

bash

# Set params on Makefile
# Setting LOCAL = False will deploy to GCP

# If the Airflow UI doesn't load, wait until the K8s dashboard is healthy (~6-10 min).
$ make start

# Once `make-start` completes, the following can be run on different terminals for monitoring.
$ kubectl get pods --watch  # to monitor the pod health
$ make browse-dash  # to get the K8s dashboard
$ make tail-k8-web  # tails log for web pod
$ make tail-k8-scheduler  # tails log for scheduler pod

# Once you are done with the services you can stop everything with the following:
$ make clean

flocasts/airflo

Airflow in Kubernetes Executor

Pre-requisite

Deployment