mapbox/dynamodb-replicator

incremental-snapshot creates extra output in snapshot

keen99 opened this issue · 0 comments

bin/incremental-snapshot.js s3://$BackupBucket/$BackupPrefix/$TABLE s3://$BackupBucket/${TABLE}-snapshot

sometimes this leaves empty lines in the snapshot output - for example:

aws s3 cp s3://$BackupBucket/${TABLE}-snapshot - | gzcat

{"what":{"S":"new1"},"a":{"S":"b"}}
{"what":{"S":"new2"},"a":{"S":"11"}}
{"what":{"S":"a"},"b":{"S":"ccd"}}
{"what":{"S":"test2"},"a":{"S":"asdf"}}
{"what":{"S":"new10"},"a":{"S":"b"}}
{"what":{"S":"sdfg"},"a":{"S":"asdf"}}
{"what":{"S":"asdf"},"aa":{"S":"bb"}}
{"what":{"S":"test"},"a":{"S":"test1"}}

{"what":{"S":"new"},"a":{"S":"fish faster 8"}}
{"what":{"S":"test4"}}
{"what":{"S":"b"},"a":{"S":"aa"},"b":{"S":"cc"}}

This one, anyway, is easy enough to handle during the uncompress ( | gzip | grep -v "^$) - but...

removed other case, found where that data came from and it was on me.