QDrant java.lang.NoSuchMethodError: 'io.grpc.MethodDescriptor$Marshaller

[AgonLohaj]

Hi,

I am using the new QDrant Spark Client (2.2.0) with the AWS EMR 7.0.0 running on Spark 3.5.0 and Java 17. When using the spark step and QDrant in EMR I run into the following issue:
java.lang.NoSuchMethodError: 'io.grpc.MethodDescriptor$Marshaller

The workaround, was to add the following at the 'shaded' section:

Is there a better way to deal with this? Let me know if you have any other questions

See the full logs below:

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Authorized committer (attemptNumber=0, stage=0, partition=0) failed; but task commit success, data duplication may happen. reason=ExceptionFailure(java.lang.NoSuchMethodError,'io.grpc.MethodDescriptor$Marshaller io.grpc.protobuf.ProtoUtils.marshaller(com.shaded.google.protobuf.Message)',[Ljava.lang.StackTraceElement;@14eec3c4,java.lang.NoSuchMethodError: 'io.grpc.MethodDescriptor$Marshaller io.grpc.protobuf.ProtoUtils.marshaller(com.shaded.google.protobuf.Message)'
at io.qdrant.client.grpc.PointsGrpc.getUpsertMethod(PointsGrpc.java:37)
at io.qdrant.client.grpc.PointsGrpc$PointsFutureStub.upsert(PointsGrpc.java:1533)
at io.qdrant.client.QdrantClient.upsertAsync(QdrantClient.java:843)
at io.qdrant.client.QdrantClient.upsertAsync(QdrantClient.java:827)
at io.qdrant.spark.QdrantGrpc.upsert(QdrantGrpc.java:36)
at io.qdrant.spark.QdrantDataWriter.doWriteBatch(QdrantDataWriter.java:71)
at io.qdrant.spark.QdrantDataWriter.writeBatch(QdrantDataWriter.java:53)
at io.qdrant.spark.QdrantDataWriter.commit(QdrantDataWriter.java:77)
at org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.$anonfun$run$1(WriteToDataSourceV2Exec.scala:459)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1409)
at org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.run(WriteToDataSourceV2Exec.scala:486)
at org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.run$(WriteToDataSourceV2Exec.scala:425)
at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:491)
at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:388)

[Anush008]

Hi @AgonLohaj. I had run into a similar issue and yes using the Shade plugin solved it for me too.
But later on, with Java 8, it worked fine as is.

So I guess, using the Shade plugin is a valid workaround.