0.0.24 doesn't process large events
Samwalton9 opened this issue · 8 comments
I'm not sure exactly where the problem is here, but I just had to downgrade to 0.0.22 (as the last version I know was working, not sure if the issue is in .23 or .24). Events that were particularly large seemed to not be coming through when looping over an EventSource feed. I'll see if I get some time to dig into this a bit further and provide more information soon.
If you can provide a test that reproduces the failure, that would be awesome.
Are you using chunked encoding by any chance? If so, I believe that this may be a duplicate of #28. Can you give https://github.com/mutantmonkey/sseclient/tree/disable_short_reads_when_chunked a try and see whether or not it fixes the issue? If so, I'll go ahead and open a PR.
Hi, I had this issue also with version 0.0.26. It seemed to be resolved when I downgraded to 0.0.22. I don't think I can paste the data here, but to provide some context the json event that it's trying to read is around 1500 lines (500KB).
@mutantmonkey I was having the problem and it was related to chunked transfer. Your code definitely works differently, compared to the latest code here in the repo. However, it doesn't seem to emit the events when the occur. For example, sometime I am getting several ping event at once, even though they usually occur with a regular frequency.
@docbobo Thanks for testing. Yeah, that would be the expected behavior as events won't be emitted until the buffer fills up. That's the problem that can be fixed by using short reads, but my understanding is that currently using chunked encoding results in errors when processing events because the chunk size is also included in the event data (see #28). The way to fix this would be to implement short reads for chunked encoding as well instead of falling back to disable them.
What's the issue you see with the current code in master?
@mutantmonkey in my case, it just completely breaks the event. No way to parse it into JSON, with the chunking not being transparently handled.
The problem with large event data is in the _event_complete method. For every new data chunk it applies a regular expression to all buffered data, resulting in a quadratic execution time.