Running pytest with local spark session
Opened this issue · 2 comments
rotemb-cye commented
Hey,
I am trying to run pytest on my local PC, when databricks extension is installed.
I am trying to create local spark session:
def get_spark_session():
spark = (
SparkSession.builder.master("local[*]")
.appName("local-tests")
.config("spark.driver.bindAddress", "127.0.0.1")
.getOrCreate()
)
return spark
@pytest.mark.etl
@pytest.fixture(scope="session")
def spark_session():
spark = get_spark_session()
yield spark
spark.stop()
and I get the following error:
RuntimeError: Only remote Spark sessions using Databricks Connect are supported. Could not find connection parameters to start a Spark remote session.
How to solve it? I want to be able to run my pytest when being offline
TNXXX
htuomola commented
Hi, this is the same exact issue that we have been struggling with. It seems that installing databricks-connect modifies installed pyspark package and adds throwing this error to the code. I'm also interested in finding a workaround for this because in the current state it basically blocks using Databricks Connect.
benoitLebreton-perso commented