SETL-Framework/setl

How to enable Hive Support?

LiuxyEric opened this issue · 2 comments

I am trying to use spark.sql to read table from hive. However Setl cannot find my table. I read the source code and noticed that when initializing the SparkSession, it didn't enable hive support. So how am I able to read my table from hive?
image
usually I just called the code below to enable hive support.

SparkSession.builder
      .enableHiveSupport
      .getOrCreate

You can either add spark.sql.catalogImplementation = "hive" into your application.conf (or other configuration files)

# src/main/resources/application.conf
setl.environment = ${app.environment}

setl.config {
  spark.app.name = "test_app"
  spark.sql.catalogImplementation = "hive"
}

or, you can use Setl.builder().setSparkSession() to provide your own spark session:

val sparkSession = SparkSession.builder().enableHiveSupport().getOrCreate()
val setl: Setl = Setl.builder()
  .setSparkSession(sparkSession)
  .withDefaultConfigLoader()
  .getOrCreate()

Thanks for the response !