[BUG] metadata-service auth type returns empty token

[ericfeunekes]

Describe the bug
Once I connect to the Databricks server from the VS Code extention, it adds .databricks/databricks.env that includes a variable DATABRICKS_AUTH_TYPE=metadata-service. I think this causes problems with authentication because I get the following error in the VS Code output window:
create_and_register_databricks_globals - ValueError: default auth: metadata-service: Metadata Service returned empty token. Config: host=https://"https://***.azuredatabricks.net/", auth_type=metadata-service, cluster_id=***, metadata_service_url=***. Env: DATABRICKS_HOST, DATABRICKS_AUTH_TYPE, DATABRICKS_CLUSTER_ID, DATABRICKS_METADATA_SERVICE_URL

To Reproduce
Steps to reproduce the behavior:

1. Follow instructions to authenticate.
2. Try to connect using Databricks connect in a jupyter notebook
3. Get the error above

System information:

1. Paste the output ot the Help: About command (CMD-Shift-P).

Version: 1.83.1 (Universal)
Commit: f1b07bd25dfad64b0167beb15359ae573aecd2cc
Date: 2023-10-10T23:46:55.789Z (2 wks ago)
Electron: 25.8.4
ElectronBuildId: 24154031
Chromium: 114.0.5735.289
Node.js: 18.15.0
V8: 11.4.183.29-electron.0
OS: Darwin arm64 22.6.0

2. Databricks Extension Version
   1.2.2

[kartikgupta-db]

Hi @ericfeunekes. This error should only be affecting local notebooks. Can you confirm that other parts of the extension are affected as well? (eg running a python file)

[ericfeunekes]

Yes that seems to be true. I don't really have a python file to run that way, but I do know it can connect to and show clusters, sync files, etc.

[Createdd]

I have the same issue. Is there any way to inspect this error in more detail? Otherwise, how to solve this namespace issue?

[ericfeunekes]

Hi @ericfeunekes. This error should only be affecting local notebooks. Can you confirm that other parts of the extension are affected as well? (eg running a python file)

This also affects the cli though. Recently I used the vs code extension and then opened a new terminal outside of vs code (in iterm2). When I tried using the cli it gave me the same error.

It seems like the extension is setting a global environment variable that needs to be unset for the cli to work.

This makes the extension break more than it helps.

[pernilak]

I have a similar problem. The extension takes over my databricks CLI. In stead of using the profile in my .databrickscfg or the configurations in my databricks.yml, it sets thing up with a default cluster (that I specified in the extension) and uses metadata-service as authentication method instead of azure cli.

[cltj]

Experiencing the same. Kindly suggest a workaround for this.

[ianepreston]

I was able to work around this issue by updating the DATABRICKS_AUTH_TYPE environment variable to match my preferred authentication. So in my case that was export DATABRICKS_AUTH_TYPE=azure_cli The terraform provider docs have a list of the possible auth types.