bradlensing/airflow

Airflow Code and Notes

Assuming airflow is installed on computer or docker container

Running Airflow Locally

In 2 seperate terminal windows run following commands

• $ airflow webserver --port 8080
• $ airflow scheduler

In browser visit localhost:8080

Running Airflow through Docker

Docker yaml file set to run in local mode

1. In airflow folder run
   • docker-compose -f docker-compose-airflow.yaml up -d
2. In VS Code Docker Tools
   • Double click on the webserver container and Attach Shell
   • May have to restart container
3. To stop running containers
   • docker-compose-airflow down

DAGs

• Demo Dag
• Monthly Interval Dag
• Tasks and Dependencies
• Using Connections, Variables and Hooks to connect an S3 bucket and log out all the sub directories
• Creating Users Table - Fetching Data - Saving to CSV - Populating to DB Dag
• S3 to Redshift DAG
  • Create Redshift Cluster Notebook (.ipynb file)
  • Documentation on Pipeline Creation