/deb-application

This repository contains application code for the Wizeline Data Engineering Bootcamp (DEB) 2023. It is one of two repositories for the DEB. The other houses the infrastructure code.

Primary LanguagePython

🗜️⚙️ DEB Application Repository 🔩🧰

Airflow Image Build Push Pyspark Scripts to GCS

This is one of two repositories with code for the entire DEB Project. While this repository focuses on the application code such as Airflow DAGs, the DEB Infrastructure repository focuses on provisioning cloud resources. This separation of concerns via separate repositories aims to follow GitOps Principles.

CI/CD Workflow

  • The build-push.yaml Github Actions workflow builds a custom airflow image and pushes it to Google Artifact Registry. It then checks out the DEB Infrastructure repository to update the airflow-helm/values-dev.yaml file with the new image and tag. For this to work, create a repository-scoped Personal Access Token for your Github account and store it as a secret in this repository. Visit Github Docs for details about creating tokens.
  • Also create Github secrets for your Github email and Username