Message size based batching
Closed this issue · 0 comments
jeffadair commented
Currently stream-producer can batch messages based on the number of records in a message. Some queuing technologies have a limit on the message size so we should guard against overflowing a message.
Since we already have a limit on the number of messages, it should load messages until it hits one limit or the other: The number of messages specified OR the size of the overall message.
SQS currently has a message limit of 256k. If we can somehow query the queue for the max message size, do that instead of hardcoding the limit, so if it changes in the future we don't have to do anything.
Barry has done something similar for the API server. Take a look at what he's done and adapt that.