databricks workspace import(_dir) should not automatically convert non-notebook .py files to notebooks
cloudyday opened this issue · 2 comments
On the Databricks platform, it is possible to use regular python modules in Repos and Workspaces.
It is even suggested by Databricks to use typical software engineering best practises. Here and here.
However, when using the Databricks CLI for importing files, especially a directory, into a Workspace by using databricks workspace import_dir, all .py files get automatically transformed to notebooks. This makes them incompatible because they cannot be imported as modules anymore.
The import and import_dir commands should probably only convert .py files starting with # Databricks notebook source to notebooks.
Glad to know I am not the only one, I opened this issue before realising you had already raised this concern.