Kafka Connect Use Case for S3 docs need to include non-default group ID configuration
osowski opened this issue · 2 comments
Multiple Kafka Connect clusters connecting to the same Kafka brokers must use unique group IDs.
The documentation in https://github.com/ibm-cloud-architecture/refarch-eda/blob/master/docs/kafka/connect-usecase-s3.md should be updated to reflect the need for a unique .spec -> config -> group.id value for the KafkaConnectS2I resources.
I was trying to test this out for a bit but the es-1 instance on the 4.3 AWS cluster seems to be wonky (the topics API seems to be dead). I tried to deploy a new Event Streams instance but the pods were giving me a "0/10 nodes are available: 4 Insufficient cpu, 6 node(s) had taints that the pod didn't tolerate."
Anyways, good news was that the sink connector initially seemed to connect, but then a few seconds later it died I believe due to the Kafka bootstrapserver not being reachable maybe?
connectorStatus: connector: state: FAILED trace: "org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches org.apache.camel.kafkaconnector.CamelSinkConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='2.4.0', encodedVersion=2.4.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='2.4.0', encodedVersion=2.4.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='2.4.0', encodedVersion=2.4.0, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='2.4.0', encodedVersion=2.4.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='2.4.0', encodedVersion=2.4.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='2.4.0', encodedVersion=2.4.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='2.4.0', encodedVersion=2.4.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='2.4.0', encodedVersion=2.4.0, type=source, typeName='source', location='classpath'}\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.connectorClass(Plugins.java:203)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.newConnector(Plugins.java:175)\n\tat org.apache.kafka.connect.runtime.Worker.startConnector(Worker.java:252)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder.startConnector(DistributedHerder.java:1183)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1400(DistributedHerder.java:125)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder$15.call(DistributedHerder.java:1199)\n\tat org.apache.kafka.connect.runtime.distributed.DistributedHerder$15.call(DistributedHerder.java:1195)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n"
Sorry, didn't get to fully resolve and test this!
Closing this issue for the group.id updates. We can sync up offline to debug environment issues.