[Issue] AzureCredentialNotFoundException: Could not find ADLS Gen2 Token
Closed this issue · 1 comments
kecheung commented
Did you read the pinned issues and search the error message?
Yes, but I didn't find the answer.
Summary of issue
I ran this code and it didn't work.
val outputContainer = "synapse"
var readDf = spark.read.format("com.microsoft.cdm")
.option("storage", "mystore12345.dfs.core.windows.net")
.option("manifestPath", outputContainer + "/dbtest2/default.manifest.cdm.json")
.option("entity", "TestEntity")
.load()
Error stack trace
23/02/09 20:01:52 ERROR Uncaught throwable from user code: com.databricks.backend.daemon.data.client.adl.AzureCredentialNotFoundException: Could not find ADLS Gen2 Token
at com.databricks.backend.daemon.data.client.adl.AdlGen2CredentialContextTokenProvider.$anonfun$getToken$1(AdlGen2CredentialContextTokenProvider.scala:30)
at scala.Option.getOrElse(Option.scala:189)
at com.databricks.backend.daemon.data.client.adl.AdlGen2CredentialContextTokenProvider.getToken(AdlGen2CredentialContextTokenProvider.scala:24)
at com.microsoft.cdm.utils.CDMTokenProvider.<init>(CDMTokenProvider.scala:15)
at com.microsoft.cdm.HadoopTables.load(HadoopTables.scala:11)
at com.microsoft.cdm.CDMCatalog.loadTable(CDMCatalog.scala:33)
at com.microsoft.cdm.CDMCatalog.loadTable(CDMCatalog.scala:15)
at org.apache.spark.sql.connector.catalog.CatalogV2Util$.$anonfun$loadTable$1(CatalogV2Util.scala:348)
at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
at org.apache.spark.sql.connector.catalog.CatalogV2Util$.loadTable(CatalogV2Util.scala:340)
at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.loadV2Source(DataSourceV2Utils.scala:136)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:323)
at scala.Option.flatMap(Option.scala:271)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:321)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:223)
Platform name
Databricks
Spark version
Spark 3.3
CDM jar version
spark3.3-1.19.4
What is the format of the data you are trying to read/write?
.csv