Handle non-idempotent Changefeed redelivery
bobvawter opened this issue · 2 comments
CockroachDB changefeeds do not guarantee an idempotent ordering of messages under all checkpointing and/or re-delivery scenarios. For some row R, it is possible to see versions of the row 1, 2a, 3, 2b, 4 (the suffix is for clarity below). While this satisfies the at-least-once delivery guarantees, it is something of a novel discovery that the system behaves this way.
This is a non-issue in fully-consistent mode, since the staging tables would suppress the duplicate 2b update.
In eventually-consistent mode (BestEffort), we bypass the staging tables whenever possible. This leads to a case where the row could roll back under the 2b message (and perhaps there is no 4 message). This could lead to rows that are desynchronized. Immediate mode is also impacted.
The proposed solution is to have BestEffort mode also write stub entries to the staging table which are applied, but that do not contain a payload. This will at least decrease the total number of bytes transferred, even if we do incur an additional put operation.
Immediate mode could also write stub entries to staging tables, but it may be more flexible if we were to add optional high-water tracking in the target database. We could then guarantee exactly-once behavior, at the cost of somewhat more expensive writes into the target database.
The lack of redelivery guarantee can be fixed if changefeed.frontier_checkpoint_frequency is set to 0.