Airflow Autoscaling ECS

Setup to run Airflow in AWS ECS (Elastic Container Service) Fargate with autoscaling enabled for all services. All infrastructure is created with Cloudformation and Secrets are managed by AWS Secrets Manager.

Requirements

Create an AWS IAM User for the infrastructure deployment, with admin permissions
Install AWS CLI running pip install awscli
Install Docker
Setup your IAM User credentials inside ~/.aws/config

    [profile my_aws_profile]
    aws_access_key_id = <my_access_key_id> 
    aws_secret_access_key = <my_secret_access_key>
    region = us-east-1

Create a virtual environment
Setup env variables in your .zshrc or .bashrc, or in your the terminal session that you are going to use:

	export AWS_REGION=us-east-1;
	export AWS_PROFILE=my_aws_profile;
	export ENVIRONMENT=dev;

Deploy Airflow Locally

make airflow-local

Deploy Airflow on AWS ECS

To deploy or update your stack run the following command:

make airflow-deploy

To rebuild Airflow Docker Image and push it to ECR (without infrastructure changes), run:

make airflow-push-image

To destroy your stack run the following command:

make airflow-destroy

Update a Dag on AWS

After creating or updating a DAG you need to rebuild Airflow image, push it to ECR and then restart the airflow service. To do all that, you just need to execute:

make airflow-push-image

Features

Control all Airflow infrastructure from a single service.yml file.
Metadata DB Passwords Managed with AWS Secrets Manager.
Autoscaling enabled and configurable for all Airflow sub-services (workers, flower, webserver, scheduler)
TODO: Continuous Integration using AWS CodePipeline
TODO: Create isolated DAGs using docker_operator

Adjust many infrastructure configs directly on Service.yml:

  workers:
    port: 8793
    cpu: 1024
    memory: 2048
    desiredCount: 2
    autoscaling:
      maxCapacity: 8
      minCapacity: 2
      cpu:
        target: 70
        scaleInCooldown: 60
        scaleOutCooldown: 120
      memory:
        target: 70
        scaleInCooldown: 60
        scaleOutCooldown: 120

Access to Airflow UI:

Look for AirflowWebServerEndpoint on outputs logged to your terminal.

    "cfn-airflow-webserver": [
        {
            "OutputKey": "AirflowWebServerEndpoint",
            "OutputValue": "airflow-dev-webserver-alb-1234567890.us-east-1.elb.amazonaws.com"
        }
    ],

Access to Flower UI:

Look for AirflowFlowerEndpoint on outputs logged to your terminal.

    "cfn-airflow-flower": [
        {
            "OutputKey": "AirflowFlowerEndpoint",
            "OutputValue": "airflow-dev-flower-alb-1234567890.us-east-1.elb.amazonaws.com"
        }
    ],

Inspired by the work done by Nicor88

ngonik/airflow-autoscaling-ecs