Client body streaming
orthecreedence opened this issue · 2 comments
Right now, the entire response (and its body) have to be read into memory before finishing the future returned from http-request. This differs from drakma in that :want-stream t will return a chunked stream with the body waiting to be read off of it...in other words, you have control over how/when the body of the response is read, as the response comes in.
I want to implement this in drakma-async. The interface will obviously have to be different. Right now, it's behaviorally similar to drakma in that :want-stream t returns a stream the body can be read from, but the idea is different: the http-stream implementation requires that the entire response be read into memory before signalling the "OK!! DONE!!" and finishing the future.
Instead there should be a way to say "return once the headers have downloaded, and alert me via this callback when new body data comes in." I tried this approach before, but had issues with chunga erroring on the stream, so I want to do something new:
Implement in cl-async tcp-stream the concept of "staging." If a stream is set to stage, data is copied off of it instead of moved off of it, and it tracks the data that's been read. You can then tell it to finalize and all the data that was copied off will be cleared out. This way, if chunga reads something it doesn't like, the error can be caught, and the data it choked on doesn't disappear, but instead still exists for the next try.
Check http-parse since it's perfect for streaming HTTP. It's worth trying to force chunga into working with async, but in case it's difficult, there's now a viable alternative.