typelevel/frameless

Seeing below Error when reading AVRO messages from Kafka Topic using Spark structured Streaming

ndhamija opened this issue · 3 comments

ndhamija commented

Seeing this Error in my Execution

Exception in thread "stream execution thread for [id = bf24558c-1749-4037-acd2-7638208464e6, runId = 6b280e35-5ee4-497c-941c-5c50b9c10f89]" java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.boxedType(Lorg/apache/spark/sql/types/DataType;)Ljava/lang/String;

More Details here :

https://stackoverflow.com/questions/62294867/problem-reading-avro-messages-from-a-kafka-topic-using-structured-spark-streamin

My Spark Shell command looks like this

spark-shell -i DataLineage.scala --jars kafka-clients-0.10.2.1.jar,spark-sql-kafka-0-10_2.11-2.3.0.cloudera1.jar,spark-streaming-kafka-0-10_2.11-2.3.0.jar,spark-avro_2.11-2.4.0-palantir.31.jar,avro-1.9.1.jar,spark-catalyst_2.11-2.4.0.jar

I guess some kind of jar's version mismatch (Mostly the 2.4.0) jars are used with spark 2.3.1, but I didn't find the respective 2.3.1 jars.

I added the spark-catalyst_2.11-2.4.0.jar with a hope, it might provide the respective classes.

imarios commented

Thanks for reporting @ndhamija! Did you try to replicate with regular spark?

ndhamija commented

you mean spark-submit ?

imarios commented

Sorry, reading this more carefully I have a different question: "how is this relevant to Frameless?"