[BUG] Magic functions no longer being executed in Interactive Window.

[cyaris]

Describe the bug
Magic functions are no longer being executed in the VS Code Interactive Window.
Running a cell using # MAGIC %md will be treated as a Python comment. The same is true for # MAGIC %run.

To Reproduce
Steps to reproduce the behavior:

1. Create a Databricks notebook source Python file.
2. Run a command in the notebook using a MAGIC function.
3. See that the MAGIC function commands are not run, and instead are rendered as comments.

Screenshots

System information:

• macOS Version 14.4.1 (23E224)
• Databricks CLI v0.216.0
• Python 3.11.0
• VS Code 1.88.1
• Jupyter v2024.3.1 (Jupyter Extension)
• Databricks v1.3.0 (Databricks Extension)
• Databricks Cluster
  • Databricks Connect Enabled
  • Databricks Runtime 13.3x

Additional context
The issue began a month or so ago.
I tried changing versions for the Jupyter and Databricks extensions but that didn't help.

[kartikgupta-db]

Hi @cyaris. Thanks for reporting this issue. It might be that there is some issue actually running our init script that sets up parsing for this magic.

• Can you try running command "Databricks: Verify Databricks notebook init scripts".
• Are you also using synapse extension by any chance? It is known to conflict with our notebooks init script.

[cyaris]

Hi @kartikgupta-db, I am not using synapse. How should the command be run?

If I run it as a string I see this...

If I run it not as a string I see this...

Thanks!

[kartikgupta-db]

I meant, running a vscode command.
On mac

• cmd + shift + p > "Databricks: Verify Databricks notebook init scripts" > enter

[cyaris]

I meant, running a vscode command. On mac

• cmd + shift + p > "Databricks: Verify Databricks notebook init scripts" > enter

Verified...

[kartikgupta-db]

Hi @cyaris. Apologies for the late response. Is this still an issue with the latest version of vscode?

[a-tanman]

Not sure if this is helpful @cyaris - I only managed to get the magic functions and line-by-line execution to work after realising that the .databricks folder in the sub-directory where the ipynb notebook was did not have .databricks.env in there, as it was created when deploying an asset bundle. I had to symlink .databricks.env which was located in the project root.

@kartikgupta-db perhaps there is a need to check that both the .databricks folder and the .databricks.env file exist?

def load_env_from_leaf(path: str) -> bool:
    curdir = path if os.path.isdir(path) else os.path.dirname(path)
    env_file_path = os.path.join(curdir, ".databricks", ".databricks.env")
    if os.path.exists(os.path.dirname(env_file_path)):

[ilia-db]

@a-tanman thanks for the pointer, that part of the code is indeed problematic, created a fix here: #1266

[ilia-db]

Published new release (1.3.1) with the fix. @cyaris can you update the extension, make sure databricks connect is enabled, and try running notebooks again?