[BUG] stdout (print & logs) is not shown when there is an error
anton164 opened this issue · 4 comments
Describe the bug
When using i.e. right-click + Upload and Run File on Databricks logs are not shown when an exception is raised. This is very frustrating as it makes for a bad debugging experience.
To Reproduce
Steps to reproduce the behavior:
- Create a file
test.py:
print("Test")
raise 2
- Right click and run on Databricks
- Debug console will only show the exception and skip any preceding logs
Hi @anton164. We are aware of this issue. Unfortunately, the way databricks backend handles command execution, makes it difficult to fetch stdout logs on errors. We are planning a workaround, where we write a file with all the stdout and stderr, and show the contents of that file on error. It will increase the latency between when the command actually finishes execution and when it prints the output. Would it be something that helps?
Thanks @kartikgupta-db that would help, but it would naturally be better to see the output in a streaming fashion. Otherwise I don't see why I would choose this over running it as a workflow which supports showing logs as they appear
@anton164 , the most common use case of this option is that it gives faster dev cycle (running as workflow has significantly more overheads). But, for longer running operations - the kind of operations for which you are likely to require streaming logs - it is indeed recommended to use workflows.
One other important advantage of workflows in this scenario is that the workflow is still running even if your connection drops out or vscode is closed. You can go back to it later through the webapp. This is not possible for runs through the "Upload and Run" functionality.
@kartikgupta-db I see thanks. I agree with the point about faster dev cycle, however logs are also very important for our dev cycle - hope to see STDOUT output integrated into "Upload and Run File on Databricks".
Thanks for your development here - overall the extension is promising!