Julia path and pkgdir must be configurable

Question

Julia path and pkgdir must be configurable

Closed this issue 7 years ago · 1 comments

The path to the Julia process, and the Julia package dir should be configurable on the nodes. We should possibly set a config variable when submitting the jobs, and use Spark's facilities to distribute that to the nodes. Then use those variables in the ProcessBuilder when starting up the Julia process.

We should NOT depend on environment variables being present on the nodes. It should happen via Spark variables.

Answer 1 · 2017-09-25T12:50:48.000Z

Fixed by #40. Set JULIA_HOME, JULIA_PKGDIR and JULIA_VERSION. For Yarn, these should be set in the Spark configuration (typically spark-defaults.conf