Question - Cluster execution on a unique cell in a .py file
KiDayz opened this issue · 6 comments
Hey,
I'm using the databricks extension in its pre-release version (v2.0.1)
I'm trying to execute a cell in a .py file (not the all file) with a cluster on Databricks (so no local kernel).
It seems that I have correclty setup the connection, my cluster is enabled & I see the 'Databricks Connect enabled' in VSCode. I thought, reading kartikgupta-db comment on this thread : #472, that I could run indiviual cell using my cluster but it seems that the only options available are the 'upload & Run file' & 'Run file as workflow'.
Am I doing something wrong here ? Or is the option not available yet ?
Many thanks !
If you have Databricks Connect enabled, then the notebooks should already be sending all the spark and dbutil commands to a Databricks cluster. We do not support executing full cell on Databricks yet.
Hey,
Thanks for your response. Do you have by any chance a roadmap or an ETA for the support of Python code execution by cell on a databricks cluster ?
I want this feature as well. E.g., when using native python to read files from UC Volums, it failes when running in VSCode as it runs locally, while it works in Databricks Workspace as it runs on the cluster. We have other examples as well - leading to our developers wanting to work in Databricks as apposed to working from VSCode.
+1 on this. This is actually one of the few blockers I have to developing solely in VSCode rather than the UI.
@kartikgupta-db - If you have a rough understanding of what would need to change for this to be implemented and would accept a PR, I'd be willing to have a go. Just need some guidance on getting started
Pluss 1 on this one!
We struggle with API calls to mlflow starts to track locally. Also when python runs locally it can't find the path to UC Volumes (of course).
Would be very nice to get this feature and a ETA as well!