Docker image and fat jar build.sbt incorrect : Missing dependencies

[jcdauchy]

I spend some times trying to use the docker image and I kept getting error on the UI, I then realized that the JAR had missing library.

So I started to try to compile a "FAT jar" but it was missing the same libraries, but sbt run was working fine.

I modified the "assembly" part in the build.sbt to get it working as follow 🔢

// Package fat jar
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = true, includeDependency = true)
/* you need to be able to undo the "provided" annotation on the deps when running your spark
programs locally i.e. from sbt; this bit reincludes the full classpaths in the compile and run tasks. */
fullClasspath in Runtime := (fullClasspath in (Compile, run)).value

assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs@_*) => MergeStrategy.discard
case _ => MergeStrategy.first
}

PS: Sorry I am not a developper at all so this might not be the best solution but I wanted a working "Fat jar" and so far it is working.

Thanks for the great work, this is an awesome project !

[roboxue]

Thanks! I realized what went wrong in docker build. I used to have spark as a direct dependency, thus java command can execute it directly. But it's no longer the case after I marked spark dependencies as provided. The simplest fix is to set docket command use sbt. I'll push a fix very soon