[BUG] Error 403 Token is expiring within 30 seconds. (token expired)

[fjakobs]

Describe the bug

The error Error 403 Token is expiring within 30 seconds. (token expired) is shown 9min after starting a "Run on Databricks" run.

To Reproduce
...

Screenshots

[fjakobs]

Our code refreshes the token when it is less than 10s before expiry. The error message suggests that the backend rejects the token earlier (30s before expiry)

https://github.com/databricks/databricks-vscode/blob/main/packages/databricks-sdk-js/src/config/Token.ts#L22

We could bump this to 30s but it might be better to improve retries instead.

[fjakobs]

Was trying to run

import time

for i in range(10):
    print(i)
    time.sleep(60)

print("Done sleeping")

and getting

20/03/2023, 17:34:59 - Creating execution context on cluster 1109-115254-ox7poobk ...
20/03/2023, 17:35:04 - Synchronizing code to /Repos/fabian.jakobs@databricks.com/ide-best-practices.ide ...
20/03/2023, 17:35:06 - Running jobs/sleep.py ...

Error: Timeout: 
Execution terminated

[fjakobs]

{
  logger: 'SDK',
  operationId: 'YYY',
  operationName: 'ClustersService.get',
  loggingFunction: 'ClustersService.get',
  timestamp: 1679402413631,
  error: {
    request: { method: 'GET' },
    response: {
      size: 0,
      [Symbol(Response internals)]: {
        type: 'default',
        url: 'https://adb-XXX.azuredatabricks.net/api/2.0/clusters/get?cluster_id=XXX',
        status: 403,
        statusText: 'Forbidden',
        headers: {
          'cache-control': 'must-revalidate,no-cache,no-store',
          connection: 'close',
          'content-length': '338',
          'content-type': 'text/html;charset=iso-8859-1',
          date: 'Tue, 21 Mar 2023 12:40:15 GMT',
          server: 'databricks',
          vary: 'Accept-Encoding',
          'x-databricks-reason-phrase': 'Token is expiring within 30 seconds. (token expired)'
        },
        counter: 0,
        highWaterMark: 16384
      }
    },
    error: {
      stack: 'Error: Error 403 Token is expiring within 30 seconds. (token expired)\n' +
        '    at ApiClient.request (/Users/fabian.jakobs/.vscode/extensions/databricks.databricks-0.3.4/out/extension.js:18708:21)\n' +
        '    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n' +
        '    at ClustersService.get (/Users/fabian.jakobs/.vscode/extensions/databricks.databricks-0.3.4/out/extension.js:20327:16)\n' +
        '    at Cluster8.refresh (/Users/fabian.jakobs/.vscode/extensions/databricks.databricks-0.3.4/out/extension.js:32261:24)\n' +
        '    at fn (/Users/fabian.jakobs/.vscode/extensions/databricks.databricks-0.3.4/out/extension.js:32302:13)\n' +
        '    at retry (/Users/fabian.jakobs/.vscode/extensions/databricks.databricks-0.3.4/out/extension.js:18403:20)\n' +
        '    at Cluster8.start (/Users/fabian.jakobs/.vscode/extensions/databricks.databricks-0.3.4/out/extension.js:32297:9)\n' +
        '    at ClusterManager.start (/Users/fabian.jakobs/.vscode/extensions/databricks.databricks-0.3.4/out/extension.js:38234:5)\n' +
        '    at ConnectionManager.startCluster (/Users/fabian.jakobs/.vscode/extensions/databricks.databricks-0.3.4/out/extension.js:38696:5)\n' +
        '    at ClusterCommands.startClusterCommand (/Users/fabian.jakobs/.vscode/extensions/databricks.databricks-0.3.4/out/extension.js:39019:5)\n' +
        '    at c.h (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:96:108776)\n' +
        '    at promptForClusterStart (/Users/fabian.jakobs/.vscode/extensions/databricks.databricks-0.3.4/out/extension.js:39267:9)\n' +
        '    at DatabricksRuntime.start (/Users/fabian.jakobs/.vscode/extensions/databricks.databricks-0.3.4/out/extension.js:39545:32)\n' +
        '    at DatabricksDebugSession.launchRequest (/Users/fabian.jakobs/.vscode/extensions/databricks.databricks-0.3.4/out/extension.js:39880:5)',
      message: 'Error 403 Token is expiring within 30 seconds. (token expired)',
      code: 403
    }
  },
  level: 'error',
  message: 'https://adb-XXX.azuredatabricks.net/api/2.0/clusters/get?cluster_id=XXX'
}

[kartikgupta-db]

@fjakobs this should be fixed now right?

[josegutierrez-db]

Hello @fjakobs @kartikgupta-db, do you know if there is any update on this issue ? Has this been fixed ?

[fjakobs]

We have a fix for this issue in https://github.com/databricks/databricks-sdk-js/pull/63.

It's not released yet but should be part of next week's release.