JS function calls fail when returning JSON is larger than 65536 bytes
Closed this issue · 3 comments
As far as I understand this is related to maximum Port buffer size but I wasn't able to find the relevant Erlang documentation.
Error example:
** (EXIT) an exception was raised:
** (Jason.DecodeError) unexpected end of input at position 65536
(jason) lib/jason.ex:78: Jason.decode!/2
(nodejs) lib/nodejs/worker.ex:39: NodeJS.Worker.decode/1
(nodejs) lib/nodejs/worker.ex:32: NodeJS.Worker.handle_call/3
(stdlib) gen_server.erl:661: :gen_server.try_handle_call/4
(stdlib) gen_server.erl:690: :gen_server.handle_msg/6
(stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Hmmmm, yeah that's definitely an issue. Thanks for reporting it. I've obviously never used this for anything with a big response.
Seems like we need to consume the response in chunks, glue them back together, and then decode. One would hope there's a newline in there already to indicate when we have a complete response, but if not then that little part is arbitrary to fix. It's reworking things to receive chunks that'll take a little doing.
I'm frequently the only person who works on this (sporadically as I need to), and I'm not sure when I'll get to it. If you don't want to wait, and taking a stab at chunking piques your interest, I'd be more than happy to review a pull request.
Off the top of my head, I'm not sure what to expect in terms of difficulty because I'm not a GenServer expert. It's either fairly straightforward because multiple concurrent chunked responses won't cross the streams (ballpark 80% chance?), or it's pretty hard because we have to figure out how to keep the right chunks in the right buffers.
(@bryanjos if you have ideas, I'd love some input.)
I don't have any thoughts except what has been mentioned. Hopefully this is a kind of problem that has been solved before and has a known work around
I might try to take a stab at this today.