Max buffer size not exceeding 262144 (Buffer overflow exception)
miratepuffin opened this issue · 5 comments
Hi all,
I have been using the library in one of my projects where a large number of akka messages are being sent around. This culminates in a large aggregate message being generated which causes a com.esotericsoftware.kryo.KryoException: Buffer overflow exception to be thrown. In response to this I have increased the buffer-size and max-buffer-size multiple times (even to max_int), but to no avail. The maximum size of this data is ~40MB, which is obviously a lot smaller than this 2GB buffer.
I have been debugging this for the last couple of hour, following the error through to the ByteBufferOutput class, and it appears that the maxCapacity never exceeds 262144 (as can be seen in the screenshot above). I can't seem to find anywhere that I could change/set this number. It also appears that the ByteBufferOutput constructor where the maxCapacity isn't passed through (line 66) is the one being called, which could be the cause of all of this.
I am using your latest version ("io.altoo" %% "akka-kryo-serialization" % "2.2.0")/akka version 2.6.14, and other than registering my classes/changing the buffer size, all the settings are as in reference conf. This can be seen here if of any assistance.
It's very much possible I am being an idiot and missed something, so if you can point me in the right direction that would be fantastic. If not and this is an issue I am more than happy to provide a more detailed run down of reproducing the problem and to help engineer a solution.
All the best,
Ben
Hi Ben
Thx for the good bug description. Nothing obvious that comes to my mind right now. We'd need to do some testing for messages that big ourselves...
Asides from that: I recommend not sending such huge messages since you'll get into lots of troubles! Clustering e.g. will start to fail (messaging blocked by one huge msg that is shared for clustermsg and user level msg). If you need to send big messages around with akka it's better to use a stream see https://doc.akka.io/docs/akka/current/stream/stream-refs.html and chunk your big message into parts. This will also give you backpressure...
Thanks for getting back to us! I apologise I should have clarified that the ~40MB is an Array of 1 million random strings that I am sending as a test case to reproduce the bug locally on my machine as it only occurs in production when we scale our system up. I think the real workloads are probably not much larger than the apparent 256KB limit, but are unfortunetly still over.
Thank you also for the akka streams link, we are intending to investigate this and see if it fits within the system.
Hi @miratepuffin
256kB should really not be any issue.
I quickly tried to reproduce your scenario 8be5fe0
but even far bigger messages work just fine... Can you write a small test that reproduces your issue (within akka-kryo-serialization)?
I looked at your application.conf. This is quiet hard to read as all the default values are in there as well. I would recommend only adding keys that you overwrite and get the other values from the respective reference.conf. This way it's easier to spot what is non standard...
Hi @danischroeter,
Thank you for going through the effort of looking into this. I have found the source of the issue and it was actually a misconfigured maximum-frame-size. I had a good delve through the conf and found that it had changed position in akka 2.6, hence why it was working before and not with my swap into artery/kryo. I apologise for the wasted time, but cheers for the advice and the rubber ducking 😄 best of luck with everything and thanks for making such a fab library.
EDIT: Just a point actually, it may be worth putting a note in the readme or your reference conf that your max-buffer-size cannot exceed akka.remote.artery.advanced.maximum-frame-size and will throw the buffer overflow exception.
All the best,
Ben
@miratepuffin Glad you found the issue 👍 and thx for the feedback!