/terraform-azure-data

Terraform script to deploy almost all Azure Data Services

Primary LanguageHCL

Azure Data Services deployment using Terraform

Description

Terraform-based deployment of the following Azure resources (default deployment settings are in the parenthesis):

  • Azure Service Bus (Standard, namespace,topic,subscription, auth. rules)
  • Azure Data Lake Storage (ZRS, Hot, Secured, StandardV2)
  • Azure Data Factory (w/Git or without)
  • Azure Data Factory linked with Data Lake Storage
  • Azure Data Factory Pipeline
  • Azure DataBricks WorkSpace (Standard)
  • Azure EventHub (Standard, namespace)
  • Azure Functions (Dynamic, LRS storage, Python, w/App.Insights or without)
  • Azure Data Explorer (Kusto, Standard_D11_v2, 2 nodes)
  • Azure Analysis Server (backup-enabled,S0, LRS, Standard)
  • Azure Event Grid (domain, EventGridSchema)
  • Azure SQL Server (version 12.0)
  • Azure SQL Database (ElasticPool, 5 GB max data size)
  • Azure SQL Elastic Pool (StandartPool, LicenseIncluded, 50 eDTU, 50GB max data size)

Content

  • auth.tf - Azure ARM provider authentication and version settings
  • main.tf - Main configuration file that describes desired Azure infrastructure
  • terraform.tfvars - Variables values to control/change the deployment settings
  • variables.tf - List of variables used in the configuration
  • outputs.tf - Outputs some useful info at the end of deployment (URLs and etc.)

Usage

Control the deployment size:

  • Open the terraform.tfvars file
  • Identify the "What Should Be Deployed?" section
  • Use true/false to setup your configuration
  • Check or change Azure service settings in the appropriate sections
  • Run terraform init to get required Terraform providers
  • Run terraform plan to initiate pre-deployment check
  • Run terraform apply to start deployment
  • (optional) terraform destroy to delete Azure resources

Tested with the latest terraform 0.15.5 and AzureRM provider 2.62.0:

Download the latest terraform: https://www.terraform.io/downloads.html

Count blocks are used frequently in this repo. If you need example of for_each block, check my gist: https://gist.github.com/rlevchenko/358ad95cc5641bc26d2a5705c2c8dbfd

Requirements

Azure Service Principal with Contributor rights or higher:

  • subscription id
  • client id
  • tenant id
  • principal secret

If Analysis Server is going to be deployed:

  • valid Azure AD user(s) UPN(s) (will be AS admin(s))

Result

Azure Resources

Terraform output