dynamodb-incremental backup

[mchhatani]

Hey,

Thanks a lot for creating this utility, it would be really helpful, if you go can go through below steps and let me know what I am missing.

I have configured the utility for taking the incremental backup for dynamodb table, I am not sure the series of steps required for successfully implement incremental backup, below are the steps I followed -

1. execute incremental-backfill, which created the single file for all items in table in s3 location.
2. enabled version control for the s3 bucket location.
3. enabled streams on dynamodb table.
4. created lambda function for capturing the update/delete/insert from the stream for the dynamodb table.
5. performed updates on few items in the table.
6. executed incremental-backfill, to take the backup again.

While executing step 6, all the items were backed again while only updated items should have been backed up.

I am not sure what should be the next step for successful implementation of the utility.

[rclark]

The incremental backfill utility is for populating the S3 bucket with the current state of your DynamoDB table. The Lambda function is responsible for ongoing inserts/updates/deletes in the S3 bucket. The ideal flow is:

1. Setup the S3 bucket, DynamoDB streams, and the Lambda function so that new changes begin being written into S3
2. Run the incremental-backfill routine once in order to populate S3 with DynamoDB records that have not been updated since the S3 bucket was created

After taking these two steps, the S3 bucket will be up-to-date and remain up-to-date as new changes are made to the DynamoDB table.

[mchhatani]

Thanks Ryan for the answer. I have configured S3 bucket with versioning, Streams, Lambda Trigger on the table.

Sorry to ask but I am new to this, do you have the code handy for lambda function (i.e. copying events to s3), I can use.

[rclark]

This library provides the JS code to run on a Lambda function to perform the incremental updates

[mchhatani]

Thanks for the response.