oysterprotocol/brokernode

[Broker] Batched uploads to s3

Opened this issue · 0 comments

https://github.com/aws/aws-sdk-go/blob/master/service/s3/s3manager/upload.go#L404

Instead of storing a JSON containing a batch, we can upload each chunk as a separate object in s3. We should mirror the logic of storing in badger as much as possible to make swapping between the different stores easy.

We should also mirror how we store chunks on the tangle so that download logic is similar.