During a resend-request the request for messages from storage allows for unbounded memory usage
philipwhiuk opened this issue · 4 comments
When a resend request to Infinity (or even just a large capped number) is made, this can cause a huge number of messages to be fetched from storage into memory which can cause the application to crash due to lack of memory. It's not practical to mitigate this in the storage layer
In addition even if they fit in memory without a fix to #271 this can lead to some fairly ugly behaviour where we continue to attempt to send messages to a session that's already disconnected us.
To Reproduce
- Send several thousand messages (volume necessary dependent on application memory profile)
- Instruct the counter-party to perform a resend-request of all these messages
Expected behavior
We should fetch the messages in batches and then send them a batch at a time
System information:
N/A
Additional context
I've got a hot fix for some of this - will try to tidy it up and submit for review.
This is probably additionally key if we implement #621
As you noted on #621 (comment) probably it would be a good thing to have a config option to restrict the maximum number of messages that could be resent. The beginning of the range could be skipped over by setting the SequenceReset tag NewSeqNo accordingly.
Yeah, a maximum resend request is probably also a good idea. I'd suggest that there's several things you could do:
- Logout the counter-party (the session is an invalid state, just like we do when the sequence number is too high
- Cap the resend amount to the limit and gap-fill
- No limit, use the batching
At some point I also need to look a throttling send - when we fixed our batching (and we really did want to send them all... we effectively DDoSed our counter-party until they caught up - ideally we could have throttled it so they stayed connected).