/terraform-databricks-examples

Examples of using Terraform to deploy Databricks resources

Primary LanguageHCLOtherNOASSERTION

terraform-databricks-examples

This repository contains the following:

  • Examples of implementing CI/CD pipelines to automate your Terraform deployments using Azure DevOps or Github Actions.

  • Multiple examples of Databricks workspace and resources deployment on Azure, AWS and GCP using Databricks Terraform provider.

There are two ways to use this repository:

  1. Use examples as a reference for your own Terraform code: Please refer to examples folder for individual examples.
  2. Reuse modules from this repository: Please refer to modules folder.

Repository structure

Code in the repository is organized into following folders:

  • modules - implementation of specific Terraform modules:
  • examples - specific instances that use Terraform modules.
  • cicd-pipelines - Detailed examples of implementing CI/CD pipelines to automate your Terraform deployments using Azure DevOps or Github Actions.

Repository content

Note
For detailed information about the examples, modules or CICD pipelines, refer to README.md file inside corresponding folder for a detailed guide on how to setup the CICD pipeline.

Examples

The folder examples contains the following Terraform implementation examples :

Cloud Example Description
Azure adb-lakehouse Lakehouse terraform blueprints
Azure adb-with-private-link-standard Provisioning Databricks on Azure with Private Link - Standard deployment
Azure adb-vnet-injection A basic example of VNet injected Azure Databricks workspace
Azure adb-exfiltration-protection A sample implementation of Data Exfiltration Protection
Azure adb-external-hive-metastore Example template to implement external hive metastore
Azure adb-kafka ADB - single node kafka template
Azure adb-private-links Azure Databricks Private Links
Azure adb-splunk ADB workspace with single VM splunk integration
Azure adb-squid-proxy ADB clusters with HTTP proxy
Azure adb-teradata ADB with single VM Teradata integration
Azure adb-uc ADB Unity Catalog Process
Azure adb-unity-catalog-basic-demo ADB Unity Catalog end to end demo including UC metastore setup, Users/groups sync from AAD to databricks account, UC Catalog, External locations, Schemas, & Access Grants
AWS aws-workspace-basic Provisioning AWS Databricks E2
AWS aws-workspace-with-firewall Provisioning AWS Databricks E2 with an AWS Firewall
AWS aws-exfiltration-protection An implementation of Data Exfiltration Protection on AWS
AWS aws-workspace-with-private-link Coming soon
AWS aws-databricks-flat AWS Databricks simple example
AWS aws-databricks-modular-privatelink Deploy multiple AWS Databricks workspaces
AWS aws-workspace-uc-simple Provisioning AWS Databricks E2 with Unity Catalog in a single apply
AWS aws-databricks-uc AWS UC
AWS aws-databricks-uc-bootstrap AWS UC
AWS aws-remote-backend-infra Simple example on remote backend
AWS aws-workspace-config Configure workspace objects
GCP gcp-sa-provisionning Provisionning of the identity with the permissions required to deploy on GCP.
GCP gcp-basic Workspace Deployment with managed vpc
GCP gcp-byovpc Workspace Deployment with customer-managed vpc

Modules

The folder modules contains the following Terraform modules :

Cloud Module Description
All databricks-department-clusters Terraform module that creates Databricks resources for a team
Azure adb-lakehouse Lakehouse terraform blueprints
Azure adb-lakehouse-uc Provisioning Unity Catalog resources and accounts principals
Azure adb-with-private-link-standard Provisioning Databricks on Azure with Private Link - Standard deployment
Azure adb-exfiltration-protection A sample implementation of Data Exfiltration Protection
Azure adb-with-private-links-exfiltration-protection Provisioning Databricks on Azure with Private Link and Data Exfiltration Protection
AWS aws-workspace-basic Provisioning AWS Databricks E2
AWS aws-databricks-base-infra Provisioning AWS Infrastructure to be used for the deployment of a Databricks E2 workspace
AWS aws-databricks-unity-catalog Provisioning the AWS Infrastructure and setting up the metastore for Databricks Unity Catalog
AWS aws-databricks-workspace Provisioning AWS Databricks E2 Workspace using pre-created AWS Infra
AWS aws-workspace-with-firewall Provisioning AWS Databricks E2 with an AWS Firewall
AWS aws-exfiltration-protection An implementation of Data Exfiltration Protection on AWS
AWS aws-workspace-with-private-link Coming soon
GCP gcp-sa-provisionning Provisions the identity (SA) with the correct permissions
GCP gcp-workspace-basic Provisions a workspace with managed VPC
GCP gcp-workspace-byovpc Workspace with customer-managed VPC.

CICD pipelines

The folder cicd-pipelines contains the following implementation examples of pipeline:

Tool CICD Pipeline
Github Actions manual-approve-with-github-actions
Azure DevOps manual-approve-with-azure-devops