Could not find ADLS Gen2 Token when running as Job

[fvalencia12]

We are creating a CDM using the 0.19 version of the connector. We use Spark context to switch the context of the running system to use an application id. When running in normal mode (not job), the code works well, but when running as a Job, we get an error stating: Could not find ADLS Gen2 Token.

We are running DataBricks Runtime 6.4 (Spark 2.4.5). With a High Concurrency cluster and Passthrough enabled.

Any help of further information would be greatly appreciated.

BTW, we have tried to use the appId and secret parameters but we get an error on that as well stating that the clientId is null.

Regards
Fabian

[bissont]

Can you provide a sample of what you mean by , "We use Spark context to switch the context of the running system to use an application id"?

What language are you using? Scala is only supported with a standard premium cluster. Does it work with a standard cluster?

I'm not aware of normal mode vs job in DB. Can you share a link? - For the appid/secrect param, does it work when using a notebook?

[fvalencia12]

What I mean by switching context I mean this, we do this so notebooks that run as jobs can actually take advantage of Passthrough: spark.conf.set("fs.azure.account.auth.type." + adlsGen2StorageName + ".dfs.core.windows.net", "OAuth") spark.conf.set("fs.azure.account.oauth.provider.type." + adlsGen2StorageName + ".dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider") spark.conf.set("fs.azure.account.oauth2.client.id." + adlsGen2StorageName + ".dfs.core.windows.net", appId) spark.conf.set("fs.azure.account.oauth2.client.secret." + adlsGen2StorageName + ".dfs.core.windows.net", secret) spark.conf.set("fs.azure.account.oauth2.client.endpoint." + adlsGen2StorageName + ".dfs.core.windows.net", "https://login.microsoftonline.com/" + tenantId + "/oauth2/token") We are using Python. What I mean by job is the job in the job blade as defined here: Jobs — Databricks Documentation<https://docs.databricks.com/jobs.html>. This allows for notebooks to be scheduled. We need to run this notebook on a schedule basis. [cid:image002.png@01D7092C.009477E0] Fabian Valencia Information Technology William Blair The William Blair Building | 150 North Riverside Plaza, Chicago, Illinois 60606 Direct: +1-312-364-5132 fvalencia@williamblair.com<mailto:fvalencia@williamblair.com> | williamblair.com<http://www.williamblair.com/> Facebook<https://www.facebook.com/WilliamBlairCo/> | LinkedIn<https://www.linkedin.com/company/166939> | YouTube<https://www.youtube.com/channel/UC7CPnCaTlZcLxOF7WFwY0uw> | Twitter<https://twitter.com/WilliamBlair> From: bissont <notifications@github.com> Sent: Wednesday, February 17, 2021 10:18 PM To: Azure/spark-cdm-connector <spark-cdm-connector@noreply.github.com> Cc: Valencia, Fabian <FValencia@williamblair.com>; Author <author@noreply.github.com> Subject: Re: [Azure/spark-cdm-connector] Could not find ADLS Gen2 Token when running as Job (#67) Can you provide a sample of what you mean by , "We use Spark context to switch the context of the running system to use an application id"? What language are you using? Scala is only supported with a standard premium cluster. Does it work with a standard cluster? I'm not aware of normal mode vs job in DB. Can you share a link? - For the appid/secrect param, does it work when using a notebook? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#67 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQZG2M46DAPFLL5LPZFTD63S7SIILANCNFSM4XYUYYUA>.

[srichetar]

You dont need to set this configuration. The CDM connector is handling internally. Please run the job in same way as you do in notebook.

[fvalencia12]

That doesn’t work because the jobs by default don’t run in a context. Meaning that they don’t run under a user or service principal. That is why when we run in interactive mode, they work as they use passthrough credentials as the user running the notebook. When we specify app id and secret we get an error. The AppId and Secret parameters would help us not to have to switch the context, but they just don’t work. Fabian Valencia Information Technology William Blair The William Blair Building | 150 North Riverside Plaza, Chicago, Illinois 60606 Direct: +1-312-364-5132 fvalencia@williamblair.com<mailto:fvalencia@williamblair.com> | williamblair.com<http://www.williamblair.com/> Facebook<https://www.facebook.com/WilliamBlairCo/> | LinkedIn<https://www.linkedin.com/company/166939> | YouTube<https://www.youtube.com/channel/UC7CPnCaTlZcLxOF7WFwY0uw> | Twitter<https://twitter.com/WilliamBlair> From: srichetar <notifications@github.com> Sent: Tuesday, February 23, 2021 9:38 PM To: Azure/spark-cdm-connector <spark-cdm-connector@noreply.github.com> Cc: Valencia, Fabian <FValencia@williamblair.com>; Author <author@noreply.github.com> Subject: Re: [Azure/spark-cdm-connector] Could not find ADLS Gen2 Token when running as Job (#67) You dont need to set this configuration. The CDM connector is handling internally. Please run the job in same way as you do in notebook. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#67 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQZG2M622UZEYXPMAM3THMTTARYAXANCNFSM4XYUYYUA>.

[fvalencia12]

Any word on this? We have a ticket open with or Premier support and going in circles with the support folks. Is AppId and Secret supported with the .18 or .19 version?

[srichetar]

Hello @fvalencia12 ,
https://docs.microsoft.com/en-us/azure/databricks/security/credential-passthrough/adls-passthrough#cluster-requirements, As per this link, Clusters enabled for credential passthrough do not support jobs

But you can continue using the CDM connector as a Job using app creds. You don't need to set the Oauth configuration. If you still face the issue, please send an email to asksparkcdm@microsoft.com and we can continue from there.

[srichetar]

Closing this as we have not heard back.