heehehe/airflow_tutorial

airflow tutorial and setting

Airflow tutorial

Apache Airflow tutorial 코드입니다.

Contents

Part	Title
1	Introduction to Apache Airflow
2	Set up airflow environment with docker
3	Set up airflow in local
4	Airflow Dag Creation Boilerplate Code
5	Build a Dag Pipeline Using Google Cloud Bigquery

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

• Clone this repo
• Install the prerequisites
• Run the service
• Check http://localhost:8080
• Done! 🎉

Prerequisites

• Install Docker
• Install Docker Compose
• Following the Airflow release from Python Package Index

Supportability

Docker Service with make cli

make init

# Build the image
# docker build . -t airflow:2.4.3

make start

# Docker compose up
# docker-compose -f docker-compose.yml up -d

make stop

# Docker compose down
# docker-compose down --volumes

Check http://localhost:8080/

User: airflow
Password: airflow

DAG Creation BoilerPlate CLI

make create

example
? dag name  movie_etl
? description  영화평점 데이터 etl 작업 dag
? owner  hwalim
? retries  3
? retry delay minutes  1
? start date (ex. 2023, 1, 1, 17, 30)  2023, 1, 1, 17, 30
? schedule interval  @daily
? catchup  False
? tags  etl