varmara/data-engineering-zoomcamp-2024

📔 Data Engineering Zoomcamp (spring 2024) by DataTalksClub - Notes, homeworks | 🛠️ GCP, BigQuery, Terraform, Docker, PostgreSQL, Mage, dbt, Spark, Kafka

Data Engineering Zoomcamp 2024 by DataTalksClub - Course Notes

This repository documents my experience with the DataTalksClub Zoomcamp data engineering course. It contains materials and assignments used to explore and apply essential concepts:

🛠️ Technologies covered

• Google Cloud Platform (GCP): Cloud-based auto-scaling platform by Google
  • Google Cloud Storage (GCS): Data Lake
  • BigQuery: Data Warehouse
• Terraform: Infrastructure-as-Code (IaC)
• Docker: Containerization
• SQL: Data Analysis & Exploration
• Mage: Workflow Orchestration
• dbt: Data Transformation
• Spark: Distributed Processing
• Kafka: Streaming

Notes and homeworks

• Module 1: Containerisation and Infrastructure as Code

  • Module 1 Homework

• Module 2: Workflow Orchestration

  • Module 2 Homework

• Workshop 1: Data Ingestion

  • Workshop 1 Homework

• Module 3: Data Warehouse

  • Module 3 Homework

• Module 4: Analytics Engineering

  • Module 4 Homework

• Module 5: Batch processing

  • Modue 5 Homework

• Workshop 2: Stream processing with RisingWave

  • Workshop 2 Homework

• Module 6: Streaming

• Weeks 7, 8, 9: Project

Resources

• Data Engineering Zoomcamp by DataTalksClub; GitHub main page: https://github.com/DataTalksClub/data-engineering-zoomcamp/?tab=readme-ov-file
• 2024 cohort playlist: https://www.youtube.com/watch?v=91b8u9GmqB4&list=PL3MmuxUbc_hKihpnNQ9qtTmWYy26bPrSb