A Harlequin adapter for Databricks. Supports connecting to Databricks SQL warehouses or Databricks Runtime (DBR) interactive clusters.
harlequin-databricks
depends on harlequin
, so installing this package will also install Harlequin.
To install this adapter into an activated virtual environment:
pip install harlequin-databricks
poetry add harlequin-databricks
If you do not already have Harlequin installed:
pipx install harlequin-databricks
If you would like to add the Databricks adapter to an existing Harlequin installation:
pipx inject harlequin harlequin-databricks
Alternatively, you can install Harlequin with the databricks
extra:
pip install harlequin[databricks]
poetry add harlequin[databricks]
pipx install harlequin[databricks]
For a minimum connection you are going to need:
- server-hostname
- http-path
- access-token
harlequin -a databricks --server-hostname my_databricks.cloud.databricks.com --http-path /sql/1.0/endpoints/1234567890abcdef --access-token dabpi***
Authentication is also possible using a username and password (known as basic authentication):
harlequin -a databricks --server-hostname my_databricks.cloud.databricks.com --http-path /sql/1.0/endpoints/1234567890abcdef --username my_user --password my_pass
Or by using OAuth user-to-machine (U2M) authentication:
harlequin -a databricks --server-hostname my_databricks.cloud.databricks.com --http-path /sql/1.0/endpoints/1234567890abcdef --auth-type databricks-oauth
For more details on command line options, run:
harlequin --help
For more information, see the harlequin-databricks Docs.
Supply the --skip-legacy-indexing
command line flag if you do not care about legacy metastores
(e.g. hive_metastore
) being indexed in Harlequin's Data Catalog pane.
This flag will skip indexing of old non-Unity Catalog metastores (i.e. they won't appear in the Data Catalog pane with this flag).
Because of the way legacy Databricks metastores works, a separate SQL query is required to fetch the metadata of each table in a legacy metastore. This means indexing them for Harlequin's Data Catalog pane is slow.
Databricks's Unity Catalog upgrade brought Information Schema, which allows harlequin-databricks to fetch metadata for all Unity Catalog assets with only two SQL queries.
So if your Databricks instance is running Unity Catalog, and you no longer care about the legacy
metastores, setting the --skip-legacy-indexing
CLI flag is recommended as it will mean
much faster indexing & refreshing of the assets in the Data Catalog pane.
Please report bugs/issues with this adapter via the GitHub issues page. You are welcome to attempt fixes yourself by forking this repo then opening an PR.
For feature suggestions, please post in the discussions.