[BUG] Stepping through python code that's running remotely on a cluster is not possible/poorly documented

[anton164]

Describe the bug
The documentation seems to imply that it's possible to step through code that's running on a Databricks cluster using the VS Code extension, however as far as I can tell this feature is only for executing spark code on a remote cluster using databricks-connect.

It would be great if you could either:

• clearly document how to step through Python code that's running on a Databricks cluster (i.e. Huggingface trainer code)
• clearly document that stepping through Python code is not supported (which I believe is the case since only pyspark code is executed remotely using databricks-connect).

[kartikgupta-db]

Hi @anton164 . This is indeed the case, that only pyspark executes remotely. But you should be able to run and debug the rest of the python code locally. I understand that this distinction is a point of confusion.
@PaulCornellDB can we maybe highlight this better in the docs?

[anton164]

@kartikgupta-db right - "you should be able to run and debug the rest of the python code locally" --> this is not really possible when you are relying on a Databricks GPU for machine learning code.

Thanks for clarifying, would love to see a way to step through code on a Databricks GPU cluster in the future!