/airflow-tutorial

Repo to learn Apache Airflow from youtube series created by tuanavu

Primary LanguageJupyter NotebookMIT LicenseMIT

Airflow tutorial

This is the code for Apache Airflow Tutorials playlist by Tuan Vu on Youtube

Contents

Part Title Git Tag
1 Introduction to Apache Airflow (blog post) v0.1
2 Set up airflow environment with docker (blog post) v0.2
3 Set up airflow environment using Google Cloud Composer (blog post) N/A
4 Writing your first pipeline (blog post) N/A
5 Airflow concept (blog post) N/A
6 Build a data pipeline using Google Cloud Bigquery (blog post) v0.6
7 Airflow variables (blog post) v0.7

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites

Usage

Run the web service with docker

docker-compose up -d

# Build the image
# docker-compose up -d --build

Check http://localhost:8080/

  • docker-compose logs - Displays log output
  • docker-compose ps - List containers
  • docker-compose down - Stop containers

Other commands

If you want to run airflow sub-commands, you can do so like this:

  • docker-compose run --rm webserver airflow list_dags - List dags
  • docker-compose run --rm webserver airflow test [DAG_ID] [TASK_ID] [EXECUTION_DATE] - Test specific task

If you want to run/test python script, you can do so like this:

  • docker-compose run --rm webserver python /usr/local/airflow/dags/[PYTHON-FILE].py - Test python script

Connect to database

If you want to use Ad hoc query, make sure you've configured connections: Go to Admin -> Connections and Edit "postgres_default" set this values:

  • Host : postgres
  • Schema : airflow
  • Login : airflow
  • Password : airflow

Credits