iobruno/data-engineering-examples

Data Engineering examples for Airflow, Prefect; dbt for BigQuery, Redshift, ClickHouse, Postgres, DuckDB; PySpark for Batch processing; Kafka for Stream processing

Data Engineering Zoomcamp

Taking the course (20205 Cohort)

• Start: 13 January 2025
• Registration link: https://airtable.com/shr6oVXeQvSI5HuWD
• Cohort folder with homeworks and deadlines

Syllabus

Module 1: Data ingestion

• Python ingestion with polars and pandas
• Rust data ingestion
• data load tool (dlt)
• IaC with Terraform (Google Cloud Platform)
• Homework

Module 2: Workflow orchestration

• Workflow orchestration with Airflow
• Workflow orchestration with Mage
• Workflow orchestration with Prefect
• Homework

Module 3: Lakehouses & Data Warehouse

• BigQuery Data Warehouse
• StarRocks Query Engine
• Lakehouse with Delta Lake
• Homework

Module 4: Analytics engineering

• BigQuery and dbt
• Redshift and dbt
• Databricks and dbt
• ClickHouse and dbt
• PostgreSQL and dbt
• DuckDB and dbt
• Data visualization with Superset/Metabase
• Homework

Module 5: Batch processing

• PySpark
• Spark + Kotlin API
• Spark (Scala)
• Homework

Module 6: Stream processing

• Stream processing with Kafka, ksqlDB and Kotlin
• Kafka Streams with ksqlDB
• RisingWave: Streaming Database
• Homework

Extras

• LakeHouse with Delta, Iceberg, Hive
• Capstone Project