Need your help in understanding below snippet, i think it should be spark-conf not config

Question

Need your help in understanding below snippet, i think it should be spark-conf not config

Closed this issue 9 years ago · 2 comments

"auxConfigGenerators" : [
{
"filename" : "config/spark-env.sh",
"sourceFilename" : "aux/client/spark-env.sh"
}

In above code, how can it refer "config" as there is no linkroot for it. I believe it should be spark-conf.
May be i am wrong but need your help to understand it.

Answer 1 · 2015-08-31T14:44:06.000Z

"filename" is the destination filename, and if you look at a History Server's configs on an actual cluster under /var/run, or through UI, you can see spark-env.sh under "config" dir along with some other files. These are referenced in control.sh as "export SPARK_CONF_DIR=$CONF_DIR/config".

Answer 2 · 2015-09-01T02:27:08.000Z

The spark client configuration is post-processed by the control.sh script, so the copy sent by the server is stored in a private location and the script then processes it and copies it to the normal /etc location.