ConduitIO/conduit-connector-kafka

Bug: Source produces records with the same position

Closed this issue · 0 comments

Bug description

Conduit requires record positions to be unique (at least records that are produced in a short time frame). The Kafka source produces records that all contain the same position (the group id), see this line (second return param is supposed to be the position).

This causes Conduit to throw this error:

2022-04-06T14:01:07+00:00 ERR encountered two records with the same position and can't differentiate them (could be that you are using a pipeline with two same source connectors and they both produced a record with the same position at the same time, could also be a badly written source connector that doesn't assign unique positions to records) component=AckerNode message_id=f8ce7c83-c0ed-4826-bc9b-8ebb2d533efb/e50b35f9-c9e0-41b7-b740-43b271107daf node_id=6967a6f8-032e-4f65-bc1a-711b1ce254a9-acker pipeline_id=d8b9a605-627e-42f0-b4e0-acaea8feaaa7

Steps to reproduce

  1. Create a pipeline with a Kafka source
  2. Create the source topic and write multiple messages in it
  3. Start pipeline
  4. It produces records with the same position and pipeline stops

Version

v0.1.0