/terraform-aws-airflow

Terraform module for a PostgreSQL-backed Apache Airflow instance

Primary LanguageHCLMIT LicenseMIT

terraform-aws-airflow

Terraform module to deploy an Apache Airflow instance on AWS backed by an RDS PostgreSQL database for storage.

Airflow architecture

How to use this Module

You can use this module from the Terraform Registry

module "example-airflow-setup" {
  source            = "unruly/airflow/aws"
  key               = "your-aws-keypair"
  db_password       = "some-db-password"
  fernet_key        = "your-fernet-key"
  vpc_id            = "some-vpc-id"
  security_group_id = "some-security-group-id"
  subnet_ids        = [ "id-1", "id-2" ]
}

Configuration

Argument Description
ami AMI code for the Airflow server (default: ami-a042f4d8 - CentOS 7 community image)
instance_type Instance type for the Airflow server (default: c3.xlarge)
key AWS SSH Key Pair name
subnet_ids List of AWS subnet ids for Airflow server and database
vpc_id AWS VPC in which to create the Airflow server
security_group_id AWS Security group in which to create the Airflow server
db_password Password for the PostgreSQL instance
fernet_key Key for encrypting data in the database - see Airflow docs

Outputs

Output Description
airflow_instance_public_dns Public DNS for the Airflow instance
airflow_instance_public_ip Public IP address for the Airflow instance
airflow_instance_private_ip Private IP for the Airflow instance
airflow_database_security_group_id Security group id for the Airflow database

Quick-start

In the ./sample directory, there is a terraform configuration file (main.tf) and an Airflow DAG file (example-dag.py).

  1. Set up Airflow in AWS eu-west-1

    terraform apply -var "key_name=<YOUR-AWS-KEYPAIR-NAME>"

  2. Wait until the webserver has started - get the url for Airflow with terraform output airflow_public_dns and navigate to it using your browser or curl

  3. Copy across the example DAG to the instance

    scp -o StrictHostKeyChecking=no example-dag.py centos@$(terraform output airflow_public_dns):/home/centos/airflow/dags

  4. Wait for Airflow to pick up the DAG and display it on the page as below

    Airflow page

  5. Congratulations, you have a running Airflow server in production!

WARNING - the database passwords and Fernet key are hardcoded in the sample configuration: do not use these in production!