airscholar/cicd_for_data_engineering

This project showcases how to integrate the world of DevOps, focusing on Continuous Integration (CI) and Continuous Deployment (CD) with the realm of modern data engineering using Terraform and Azure as the case study

This repository is an Infrastructure as Code (IaC) project that leverages terraform for orchestrating the provisioning of the Azure services related to Data Engineering.

Prerequisites

1. Azure Account with a subscription
2. Azure CLI
3. Terraform CLI
4. Any IDE of your choice (VS Code, PyCharm, etc.)

System Architecture

Getting Started

1. Clone this repository
2. Replace the content of the file named terraform.tfvars in the root directory of the project
3. Run the following commands in the root directory of the project:
   • terraform init (to initialize the terraform project)
   • terraform validate (to validate the terraform code)
   • terraform plan (to see the changes that will be applied)
   • terraform apply (to apply the changes)
   • terraform destroy (to destroy the infrastructure)

Video