urakozz/go-dynamodb-stream-subscriber

checkpointing for streams

Opened this issue · 2 comments

abhineshada commented
  • Common approach to handle checkpointing while using GO sdks?
  • How to handle missing/duplicated data?
urakozz commented

What the hell is checkpointing? I didn't use DynamoDB for some years, is is new term there?

How to handle missing/duplicated data?

However you like, this library only wraps complexity of iterating over overengineered dynamo streams

abhineshada commented

By checkpoint, I mean tracking the status of shards/data processed.

Refer: https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-ddb.html
https://www.infoq.com/articles/demystifying-dynamodb-streams

KCL adapter for dynamoDB streams has checkpoint feature (only available in Java): https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.KCLAdapter.html

The library is really well written. I had few doubts regarding how data accuracy can be maintained using go SDK.