jonathanneo/AzureDataPlatform

The Telstra Purple Azure Data Platform. Keeping a copy to show to my future offsprings - "A relic of the past".

Azure Data Platform

The Azure Data Platform allows you to quickly spin up an end-to-end best practice environment that connects the various Azure Data Services resources together so that you can quickly and securely create ETL jobs using Data Factory, Databricks, Synapse, Purview, Data Lake, etc.

Superseded in July 2022

This data platform was retired from usage by Telstra Purple at the start of July 2022 due to improvements made by Microsoft on their Cloud Adoption Framework. Check out the following links before deciding to use this platform:

• https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/cloud-scale-analytics/architectures/data-management-landing-zone
• https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/cloud-scale-analytics/architectures/data-landing-zone
• https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/cloud-scale-analytics/architectures/what-is-data-mesh

Documentation

• Infrastructure
  • Configuration
  • Limitations
  • Optional
  • Pipeline
  • Post Deployment Steps
• Databases
• Databricks
• Data Factory
• Web App
  • WebApp Settings
  • User Guide
• Function Application

Prerequisites

Firstly you need to make sure your local environment meets these pre-requisites (you can use Windows, Mac or Linux):

• VS Code - run choco install vscode on Windows with Chocolatey
• .NET Core 3.1 SDK and .NET Core 2.1 runtime
• PowerShell 7.1 (can be installed side-by-side with PowerShell 5.1) - run choco install powershell-core on Windows with Chocolatey. You will need at least 7.1.0.
• Az PowerShell module at least 5.1.0 - run Install-Module -Name Az -Force -Verbose in an admin console
• DatabricksPS PowerShell module at least 1.6.2.0 - run Install-Module -Name DatabricksPS -RequiredVersion 1.6.2.0 -Force -Verbose in an admin console
• azure.datafactory.tools PowerShell module at least 0.17.0 - run Install-Module -Name azure.datafactory.tools -RequiredVersion 0.17.0 -Force -Verbose
• Azure Functions Core Tools - run choco install azure-functions-core-tools-3 on Windows with Chocolatey
• Az.Synapse PowerShell Module at least 0.17.0 - run Install-Module -Name Az.Synapse -Requiredversion 0.17.0
• Az.Purview PowerShell Module at least 0.1.0 - run Install-Module -Name Az.Purview -Requiredversion 0.1.0

License

GNU GENERAL PUBLIC LICENSE - Version 3, 29 June 2007