JahstreetOrg/spark-on-kubernetes-helm

Question: Livy Scala Api

Opened this issue · 1 comments

What version of scala/spark do client applications need to use?
The image says Spark 3.0.1 but Livy api requires Scala 2.11 which only works with earlier versions of Spark.

I am getting a ClassNotFound error when trying to submit a job:
00:20 WARN: [kryo] Unable to load class org.apache.livy.scalaapi.LivyScalaClient$$anon$1 with kryo's ClassLoader. Retrying with current..
21/08/23 19:14:12 INFO JobWrapper: Failed to run job b3353a3c-2140-4d4c-94c2-520473152ec2
org.apache.livy.shaded.kryo.kryo.KryoException: Unable to find class: org.apache.livy.scalaapi.LivyScalaClient$$anon$1

Hi @jdcohen220 , sorry for the late reply, getting back to this project only now 🙏 .

Is it still an issue for you? Happy to guide you through the resolution in case you provide me the details on reproducing it.