[Feature request] set working directory when running as workflow

[nickstanishadb]

A common pattern in my team is to start every repo notebook with %pip install -r ../../../requirements.txt. A minor annoyance I have using the VSCode extension Run As Workflow On Databricks is that this doesn't work out of the box. I suspect it's because a new notebook gets materialized to run the workflow and it's not necessarily in the same path as where I'm developing.

Could we add a step to the preamble that does os.chdir("/abolute/path/to/synced_notebook_dir")? That way when I do Run As Workflow on Databricks not only am I running the code, but I am also running it as if I was working in the notebook's remote directory?

[mroy-seedbox]

Related: What is the default current working directory in Databricks Runtime 14.0 and above?

This change brings notebook interaction with workspace files in line with behavior observed in Databricks Repos.

I suspect the VS Code extensions works the same as Databricks Repo, not workspace files.

[kartikgupta-db]

@mroy-seedbox the extension indeed works the same as repos. We try to maintain that parity because workspace fs is relatively newer and we have a lot of users on the older repos.

We have some preamble code to make the behaviour of workspacefs similar to repos for all dbrs. This issue requires a small modification to that preamble code.