codemancers/rbkit

Split event collection messages and send smaller sized messages

emilsoman opened this issue · 0 comments

When a large number of messages keep accumulating, or when objectspace dump is generated, too much memory is realloc'ed and also a huge chunk of data is sent over zmq.

We should keep accumulating messages, but :

  1. Keep a limit on message queue, and send out the messages when this limit is reached.
  2. Even if the limit is not reached, send whatever's there every second.
  3. If a message will not fit inside this message queue, split the payload into multiple messages(of same timestamp), and indicate that the payload is sent in parts.