A collection of DynamoDB helpers written in Golang to assit with reading and writing data.
go get github.com/samit22/go-ddb
To get maximum read throughput from a DynamodDB table we can leverage the Parallel Scan functionality.
// structure of DDB item
type message struct {
name string `json:"name"`
}
// new scanner with table name and total segments
scanner := ddb.NewScanner(ddb.Config{
TableName: "ddb-table-name",
TotalSegments: 150, // calculate value: (table size GB / 2GB)
SegmentOffset: 0, // optional param for controlling offset
SegmentCount: 150, // optional param for controlling how many routines get created
})
// start parallel scan w/ handler func
scanner.Start(ddb.HandlerFunc(func(items ddb.Items) {
for _, item := range items {
var msg message
dynamodbattribute.UnmarshalMap(item, &msg)
fmt.Println(msg.Name)
}
}))
// wait for all scans to complete
scanner.Wait()Leverage a checkpoint table to store the last evaluated key of a scan:
scanner := ddb.NewScanner(ddb.Config{
TableName: "ddb-table-name",
CheckpointTableName: "checkpoint-production", // name of table to store last evaluated keys
CheckpointNamespace: "my-sample-app", // namespace to avoid collisions with other scripts
TotalSegments: 150,
})