Review Setl builder
Closed this issue · 1 comments
maroil commented
DCContext builder should be able to parse spark related configuration from the config file when setDCContextConfigPath
is called.
In a section like the following:
context_config {
spark {
master_url = "url"
app_env = ""
}
}
qxzzxq commented
Setl will look by default the following configuration in the config file:
setl.config {
spark {
spark.default.parallelism = "200"
spark.sql.shuffle.partitions = "200" # set a bigger value if memory spills
}
}