Run time java.lang.NoSuchMethodError for OpenChannelRequestBuilder.setOffsetTokenVerificationFunction

[PasinduDissMrYum]

Hi there,
We're running the snowflake-kafka-connector via MSK connect and seeing an unrecoverable error java.lang.NoSuchMethodError: 'net.snowflake.ingest.streaming.OpenChannelRequest$OpenChannelRequestBuilder net.snowflake.ingest.streaming.OpenChannelRequest$OpenChannelRequestBuilder.setOffsetTokenVerificationFunction(net.snowflake.ingest.streaming.OffsetTokenVerificationFunction) could someone please provide some insight into why this could be happening ?

Connector Configuration:

name=snowflake-connect
connector.class=com.snowflake.kafka.connector.SnowflakeSinkConnector
tasks.max=4
topics=data-events.v1
buffer.count.records=10000
buffer.flush.time=5
buffer.size.bytes=20000000
errors.log.enable=true

snowflake.url.name=<account-id>.snowflakecomputing.com:443
snowflake.ingestion.method=SNOWPIPE_STREAMING
snowflake.topic2table.map=data-events.v1:data_events
snowflake.user.name=<SNOWFLAKE_USER>
snowflake.private.key=<PRIVATE_KEY>
snowflake.database.name=<DATABASE_NAME>
snowflake.schema.name=<SCHEMA_NAME>
snowflake.role.name=<ROLE_NAME>
snowflake.enable.schematization=true

key.converter=org.apache.kafka.connect.storage.StringConverter
key.converter.schemaAutoRegistrationEnabled=false
key.converter.schemas.enable=false

value.converter=com.amazonaws.services.schemaregistry.kafkaconnect.protobuf.ProtobufSchemaConverter
value.converter.endpoint=https://glue.ap-southeast-2.amazonaws.com
value.converter.region=ap-southeast-2
value.converter.dataFormat=PROTOBUF
value.converter.registry.name=<REGISTRY_NAME>
value.converter.schemaName=data-events-v1
value.converter.schemas.enable=true
value.converter.schemaAutoRegistrationEnabled=true

Plugins zip:
Plugins zip consists of snowflake version 2.2.2 and AWS Glue protobuf converter version 1.1.19

Snowflake Connector v2.2.2
AWS Glue protobuf converter v1.1.19

Full stack trace

[Worker-0c6738eb5b4a2cde7] at java.base/java.lang.Thread.run(Thread.java:829)
[Worker-0c6738eb5b4a2cde7] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[Worker-0c6738eb5b4a2cde7] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[Worker-0c6738eb5b4a2cde7] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[Worker-0c6738eb5b4a2cde7] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:238)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:189)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:232)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:324)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.connect.runtime.WorkerSinkTask.pollConsumer(WorkerSinkTask.java:457)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1226)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1257)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:508)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:365)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:449)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:430)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.invokePartitionsAssigned(ConsumerCoordinator.java:293)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.connect.runtime.WorkerSinkTask$HandleRebalance.onPartitionsAssigned(WorkerSinkTask.java:705)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.connect.runtime.WorkerSinkTask.access$1100(WorkerSinkTask.java:71)
[Worker-0c6738eb5b4a2cde7] at org.apache.kafka.connect.runtime.WorkerSinkTask.openPartitions(WorkerSinkTask.java:640)
[Worker-0c6738eb5b4a2cde7] at com.snowflake.kafka.connector.SnowflakeSinkTask.open(SnowflakeSinkTask.java:262)
[Worker-0c6738eb5b4a2cde7] at com.snowflake.kafka.connector.internal.streaming.SnowflakeSinkServiceV2.startPartitions(SnowflakeSinkServiceV2.java:223)
[Worker-0c6738eb5b4a2cde7] at java.base/java.lang.Iterable.forEach(Iterable.java:75)
[Worker-0c6738eb5b4a2cde7] at com.snowflake.kafka.connector.internal.streaming.SnowflakeSinkServiceV2.lambda$startPartitions$1(SnowflakeSinkServiceV2.java:228)
[Worker-0c6738eb5b4a2cde7] at com.snowflake.kafka.connector.internal.streaming.SnowflakeSinkServiceV2.createStreamingChannelForTopicPartition(SnowflakeSinkServiceV2.java:260)
[Worker-0c6738eb5b4a2cde7] at com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel.<init>(TopicPartitionChannel.java:275)
[Worker-0c6738eb5b4a2cde7] at com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel.openChannelForTable(TopicPartitionChannel.java:1063)
[Worker-0c6738eb5b4a2cde7] java.lang.NoSuchMethodError: 'net.snowflake.ingest.streaming.OpenChannelRequest$OpenChannelRequestBuilder net.snowflake.ingest.streaming.OpenChannelRequest$OpenChannelRequestBuilder.setOffsetTokenVerificationFunction(net.snowflake.ingest.streaming.OffsetTokenVerificationFunction)

[sfc-gh-wfateem]

@PasinduDissMrYum the exception java.lang.NoSuchMethodError typically means there's a mismatch between library versions on the classpath somehow. So for this error:

java.lang.NoSuchMethodError: 'net.snowflake.ingest.streaming.OpenChannelRequest$OpenChannelRequestBuilder net.snowflake.ingest.streaming.OpenChannelRequest$OpenChannelRequestBuilder.setOffsetTokenVerificationFunction(net.snowflake.ingest.streaming.OffsetTokenVerificationFunction)

That's saying the class OpenChannelRequestBuilder that was loaded at runtime didn't have the method setOffsetToken. This method was added in the Snowflake Ingest Java SDK in v2.1.0, and the Snowflake Kafka Connector starts using that functionality in v2.2.1.

It would then suggest that somehow, you ended up having a Snowflake Ingest Java SDK library version older than v2.1.0 on the classpath that could be causing the issue.

Can you take a look and see what might be adding another version of the snowflake-ingest-sdk-<version>.jar library on your classpath? If you can't figure that out, then you'll want to add the following JVM argument to debug the issue and see where the class OpenChannelRequestBuilder is being loaded from:
-versbose:class

[sfc-gh-wfateem]

@PasinduDissMrYum have you been able to figure this out?

[sfc-gh-wfateem]

@PasinduDissMrYum I'm going to close out the issue for now. Feel free to reopen it if you still need help with this.

[PasinduDissMrYum]

Thank you @sfc-gh-wfateem, the issue was due to a misconfigured plugin.