Large file uploads
Closed this issue · 12 comments
Is it possible to handle large file uploads in middleware, i.e. stream the body to disk instead of keeping it in memory ?
There is currently no streaming or chunked support for receiving requests, only chunked transfer responses.
The reason is to make handlers and middlewares semantically simpler. When the handler is called, it knows that whatever is inside of the request will not change. The return value from the handler is what is returned to the user. This avoids many of the problems found in Cowboy, Misultin, Mochiweb and Yaws. For example every time you look at a part of the request it might have side effects like talking to a port or asking a process for some data. Whenever you send a response, great care needs to be taken that you don't send the body before the headers, or the body twice, etc, which again might be implemented with messaging a port or another process.
Elli makes a tough choice to have simple semantics of writing a handler. The handler can be referentially transparent and without side-effects, at least as far as dealing with the request and response is concerned.
With that said, maybe there would be a way of allowing people to venture into crazy land if they so wish. I have been playing around with handing over the socket to the handler after receiving the headers. The handler can then decide what behaviour fits this particular request best. Maybe the handler wants to receive the body in multiple chunks, stream it to disk, implement websockets, etc. Specifically, I'm thinking maybe there would be an "init" function called inside the handler which gets the request without the body and can signal the behaviour wanted by returning a value. If the function is not defined, the default of "rack-style" is used.
I know of some more users that has been wishing for the same feature which would allow them to implement websockets. Do you have any more use cases to add to the list?
I can't think of any case except file uploads and websockets.
If it's possible to delegate this functionality to middleware while keeping the core simple, i think it would make a great addition to Elli - in the spirit of "simple things should be simple, complex things should be possible"...
For my use case, I can upload large files directly to S3 and handle the rest with Elli.
I will give implementing it a go soon. Then we can see what it would look like.
That's great.
I would try to implement the disk streaming when you have done an init callback.
Check out the "handover" branch. If you export an init/2 function in your handler and return {ok, handover}, Elli will call your handle/2 before receiving the body. You can then receive the body and send the response on your own using elli_http:send_response/2. The return value should be {close | keep_alive, Buffer :: binary()} where Buffer is any data belonging to the next request, otherwise empty binary.
I'll try the handover approach over the weekend and give you feedback. I did some work on multipart parsing - https://github.com/mme/erlmultipart
This is a working sample that streams the upload to disk: https://github.com/mme/elli_upload_sample
Is it possible to go back to the normal handle flow after body parsing ?
Hey, sorry for not getting back to you. I started a new job and it has been a bit crazy.. :-)
This stuff looks really cool. I'll try to clean up the handover branch a bit soon and merge it to master. I will also give a go at going back to the normal handle flow.
Here is elli_websocket, also build on the handover branch. https://github.com/mmzeeman/elli_websocket
Any chance of this being included? elli would be a nice successor to mochiweb and this seems like the main missing feature.
Isn't this already included? Seems like handover functionality is in master https://github.com/knutin/elli/blob/master/src/elli_example_callback_handover.erl
Yup, it's been there a while even, just forgot to close this ticket. (: