/airflow_for_beginners

Primary LanguagePythonMIT LicenseMIT

This is a small example of the workflow built with Apache Airflow.

You can find slides here and watch the talk here

The goal is to set up a data pipline to get a fresh portion of Stack Overflow questions with tag pandas to our mailbox daily.

A small python script could do the job, but for the learning purposes we choose to overengineer it.

By writing this workflow we will learn the main concepts of Apache Airflow, such as:

  • Operators
  • DAG
  • Tasks
  • Hooks
  • Variables
  • Connections
  • XComs

Happy learning 🤓

Helpful resources

📝 Apache Airflow Documentation

Apache Airflow tutorials for beginners

📝 Apache Airflow Tutorial for Data Pipelines

📝 Apache Airflow for the confused

📝 Airflow: Tutorial and Beginners Guide

📝 ETL Pipelines With Airflow

Some more

📰 ETL best principles

📰 Managing Dependencies in Apache Airflow

📝 Getting Started with Airflow Using Docker

🎧 Putting Airflow Into Production

📝 How to configure SMTP server for apache airflow

If you have any questions or would like to get in touch with me, please drop me a message to hello@varya.io