Task fails with: java.lang.NoClassDefFoundError: io/confluent/connect/avro/AvroData
Symbianx opened this issue · 2 comments
Hello, first of all thank you for the amazing work put into this connector :)
We've been trying to get this the connector to work with files from S3, these files were pushed there with the data formatted in Avro format by the Confluent S3 Sink connector.
We were able to get the connector to talk to S3 and it finds the files there, however we're getting the following error now. Looks like some jar is missing from the classpath.
[2020-03-23 15:55:37,167] ERROR Error reading file from FS: s3a://some-bucket/topics/some-file+0+0000000236.avro. Keep going... (com.github.mmolimar.kafka.connect.fs.FsSourceTask:80)
org.apache.kafka.connect.errors.ConnectException: An error has occurred when creating reader for file: s3a://some-bucket/topics/some-file+0+0000000236.avro
at com.github.mmolimar.kafka.connect.fs.policy.AbstractPolicy.offer(AbstractPolicy.java:208)
at com.github.mmolimar.kafka.connect.fs.policy.SleepyPolicy.offer(SleepyPolicy.java:11)
at com.github.mmolimar.kafka.connect.fs.FsSourceTask.lambda$poll$0(FsSourceTask.java:74)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at com.github.mmolimar.kafka.connect.fs.FsSourceTask.poll(FsSourceTask.java:71)
at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:244)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:220)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: io/confluent/connect/avro/AvroData
at com.github.mmolimar.kafka.connect.fs.file.reader.AvroFileReader$GenericRecordToStruct.<init>(AvroFileReader.java:107)
at com.github.mmolimar.kafka.connect.fs.file.reader.AvroFileReader.<init>(AvroFileReader.java:32)
at sun.reflect.GeneratedConstructorAccessor51.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.github.mmolimar.kafka.connect.fs.util.ReflectionUtils.make(ReflectionUtils.java:31)
at com.github.mmolimar.kafka.connect.fs.util.ReflectionUtils.makeReader(ReflectionUtils.java:19)
at com.github.mmolimar.kafka.connect.fs.policy.AbstractPolicy.offer(AbstractPolicy.java:205)
We built the connector with:
mvn package -f /tmp/connect-fs-plugin/kafka-connect-fs-0.2
Copied the whole folder /tmp/connect-fs-plugin/kafka-connect-fs-0.2/target/kafka-connect-fs-0.2-package/share/java/kafka-connect-fs into the plugins folder of kafka-connect, shouldn't that include all the dependencies?
Hi @Symbianx and many thanks!
In that version, the dependency io.confluent:kafka-connect-avro-converter is set as provided. So, if you repackage it removing <scope>provided</scope> in the pom.xml you'll get the jar which is missing (I guess you're not using the Confluent dist).
Currently I'm working on a new version with several improvements, policies, new file readers and also the upgrade to the latest Kafka version. It's on develop branch right now (snapshot) but I'll release a new version in few weeks. So, I'd say it could be even better to use this snapshot.
Thanks for the fast response!
After some tries we got it running with the pom.xml change.
I did see you're introducing a new JSON file reader but we need this now and since it's for a critical component we'd rather stay with the stable release. :)