KPL sent records out of order
loneknightpy opened this issue · 2 comments
KPL sent records of out order even when maxConnections was set to 1 and no error/retry occurred. I cannot find detail of KPL order guarantee in the documents. Is there a way to ensure ordering of records sent from KPL?
Kinesis in general does not guarantee "write ordering". It does guarantee "read order", in that records successfully written to Kinesis are guaranteed to be always read out in the same order.
Even if you are not using the KPL and used the SDK directly, records are not guaranteed to be lodged in Kinesis in the same order they are written to the SDK, for a variety of reasons related to the way distributed systems work. Because the KPL uses the putRecords web API, it has the same side effect.
The only way to guarantee "write order" is if you have only one producer, and you use the "putRecord" (not putRecords) API of the SDK one record at a time, and use the "sequenceNumberForOrdering" feature. Basically you will put record n only after getting a successful response to the put for record n-1.
Hope this helped.
@gharikum Got it, thanks for the clarification.