SETL-Framework/setl

Review Setl builder

Closed this issue · 1 comments

DCContext builder should be able to parse spark related configuration from the config file when setDCContextConfigPath is called.

In a section like the following:

context_config {
  spark {
    master_url = "url"
    app_env = ""
  }
}

Setl will look by default the following configuration in the config file:

setl.config {
  spark {
    spark.default.parallelism = "200"
    spark.sql.shuffle.partitions = "200"  # set a bigger value if memory spills
  }
}