- Unity Catalog enabled Workspace
- DBR 13.0 and above
- Python version (minor version must match between the cluster and your local environment)
- The Databricks Connect major and minor package version should match your Databricks Runtime version.
-
Use Python Virtual Environment (venv or conda). I prefer conda. See comparision here
-
Existing pyspark on your local machine will conflict with Databricks Connect pyspark. You must uninstall before you install databricks connect.
-
For verbose logging and debugging databricks connect issues,
export SPARK_CONNECT_LOG_LEVEL=debug
- Step 1 - Setup Python virtual environment on your local machine
Important
- Local Python version should match with the DBR Python version. Run (%sh python --version in your cluster)
- The Databricks Connect major and minor package version should match your Databricks Runtime version (e.g. DBR 13.0 cluster requires databricks-connect=13.0.* package)
$ conda create --name dbconnectv2 python=3.10
$ conda activate dbconnectv2
- Step 2 - Install Databricks Connect Client
First, confirm there is no existing pyspark. if found, run pip3 uninstall pyspark
pip3 show pyspark
WARNING: Package(s) not found: pyspark
Then, install databricks-connect package
$ pip3 install --upgrade "databricks-connect==13.0.*"
# --upgrade option upgrades any existing client installation to the specified version.
- Step 3 - Configure Connection Properties
You'll need:
-
Workspace URL
-
Databircks PAT
-
Cluster ID
-
Cluster Port - typically 443.
- Step 3.1 - create a databricks configuration profile file
.databrickscfg.
in your home directory
$ touch ~/.databrickscfg # If you are a databricks CLI user, you can skip this step as you'll already have this file.
- Step 3.2 - add the configuration in the configuration profile file. Example configuration is shown below:
[dbconnect-dev] host = https://<workspace-instance-name> token = <access-token-value> cluster_id = <cluster-id>
- Step 3.1 - create a databricks configuration profile file
-
Step 4 - Initialize
DatabricksSession
Create a python file and initialize the DatabricksSession
as follows:
touch hello_databricks_connect.py
from databricks.connect import DatabricksSession
from databricks.sdk.core import Config
config = Config(profile = "dbconnect-dev")
spark = DatabricksSession.builder.sdkConfig(config).getOrCreate()
- Step 5 (optional) - Set the environment variable You only need to do this if you don't use the configuration profile or you want to use the pyspark console for remote execution.
export SPARK_REMOTE="sc://<WORKSPACE_URL>:443/;token=<TOKEN>;x-databricks-cluster-id=<CLUSTER_ID>"
- Step 6 - Test the connectivity
$ pyspark
Python 3.10.11 (main, Apr 20 2023, 13:59:00) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 13.0.0
/_/
Using Python version 3.10.11 (main, Apr 20 2023 13:59:00)
Client connected to the Spark Connect server at xxxxxxxx.cloud.databricks.com:443
SparkSession available as 'spark'.
>>> spark.range(10).show()
+---+
| id|
+---+
| 0|
| 1|
| 2|
| 3|
| 4|
| 5|
| 6|
| 7|
| 8|
| 9|
+---+
Steps to setup a a VS Code project
- create a project dir:
$ mkdir hello_databricks_connect
- Activate python virtual environment containing databricks-connect package:
$ conda activate dbconnectv2
- Open the project directory in VS Code:
$ code .
- Set the current Python interpreter to be the one that is referenced from the virtual environment. On the Command Palette (View > Command Palette), type Python: Select Interpreter, and then press Enter.
- create a
hello_databricks_connect.py
file with the following code:
from databricks.connect import DatabricksSession
from databricks.sdk.core import Config
config = Config(profile = "dbconnect-dev")
spark = DatabricksSession.builder.sdkConfig(config).getOrCreate()
df = spark.read.table("samples.nyctaxi.trips")
schema = df.schema
all_columns = [col.name for col in schema]
number_of_columns = len(all_columns)
numeric_columns = [col.name for col in schema if col.dataType.typeName() in ["integer", "double", "float", "long", "decimal"]]
longer_trips = df.filter(df.trip_distance > 10)
longer_trips_count = longer_trips.count()
print("Total number of columns: " + str(number_of_columns))
print("Total number of longer trips: " + str(longer_trips_count))
- Click Run
Debugging
- With the Python code file open, set any breakpoints where you want your code to pause while running.
- Click the Run and Debug icon on the sidebar
- Select the Python file you want to debug
For VS Code specific Python code debuging insgtructions, see here
Databricks Connect V2 documentation Databricks Connect V2 Release Notes
How to get connection details for your Databricks cluster
Understanding Workspace URL - Instance and ID
Understanding Cluster URL - ID