Errors when running the terraform

[vslegers]

Hello Alex,

First of all, thanks for the interesting demo! I'm currently working on CICD for DLT pipelines.

I was trying to go through the demo but ran into issues when executing the terraform code (attached screenshot) and not sure how to proceed. Could you please have a look and offer some guidance?

Thanks in advance!

Kind regards,
Vincent

[alexott]

You need to use one of the authenticaiton methods as it's described in the terraform provider documentation. I'll update README.

[vslegers]

Hello Alex, thanks for the quick response!

I managed to run largest part of the terraform code now. I see the resources have been created in Databricks (dev/staging/prod repos and DLT pipeline(s)).

Also the Azure DevOps project was deployed but I believe there is still an issue because I cannot see the Build Pipeline or Repo in the project. Can this be connected to error I received at the end of terraform execution?

I tried manually creating and running the Build Pipeline from the YAML in the repo but also get an error on Variable group not being found.

Can you offer some guidance on how I can mitigate these issues? Thanks a lot in advance.

Kind regards,
Vincent

[alexott]

if you do it manually, you need to create a variable group with name DLT Files In Repos Testing and populated it with all variables listed here: https://github.com/alexott/dlt-files-in-repos-demo/blob/main/terraform/azuredevops/devops.tf#L47

[vslegers]

Ok, I will try that. Thanks again for the quick response!

[SolitanReiska]

Hi Alex, thanks for the repo. concerning the auth method here, I suggest adding a default authentication method to main.tf with databricks_url and databricks_pat and

• add that option into the provider block of main.tf

 provider "databricks" {
  host  = var.databricks_workspace_url
  token = var.databricks_pat
}

• to the variable.tf

variable "databricks_workspace_url" {
  description = "Databricks URL"
  type        = string
}

variable "databricks_pat" {
  description = "Databricks PAT"
  type        = string
}

and then add it to the README.md with a hint that this might not work for everybody. When prototyping it is often on a sandbox workspace with PAT. Then this option is the default. Anybody familiar with service principals and other auth methods will know what to do. But this way everything will run out of the box in the most simple scenario?

[alexott]

It’s mentioned in the readme that you need to configure authentication. The suggested method doesn’t work for my workflows, as for me it’s easier to specify environment variable pointing to a given profile than to change parameters…

[SolitanReiska]

That's what I mean. A pro like you knows either way what to do and can quickly remove parameter from the provider block, but somebody not so familiar with this flow has everything rigged without studying TF auth methods.

[SolitanReiska]

Or maybe add that case as example to the README.md

Authentication example with Databricks personal access token:

1. Add additional variables to the terraform.tfvars
databricks_url - URL of your existing Databricks Workspace, like, https://adb-3946431527987256.16.azuredatabricks.net
databricks_pat - Databricks personal access token (PAT) obtained as described in documentation.
2. Add the variables to variables.tf

 variable "databricks_workspace_url" {
  description = "Databricks URL"
  type        = string
}

variable "databricks_pat" {
  description = "Databricks PAT"
  type        = string
}

3. Add the variable reference into the databricks provider block in main.tf like

provider "databricks" {
  host  = var.databricks_workspace_url
  token = var.databricks_pat
}