Feature Request: Ability to stream blob responses

Question

Feature Request: Ability to stream blob responses

dominicbarnes opened this issue 12 years ago · 9 comments

I'd like to be able to act as a pseudo-proxy for S3 and pipe a readable stream of my S3.GetObject request directly to my end-user, particularly if it's something like an image where it's just blob data, especially if I don't plan on manipulating the object itself. Currently, I have to wait for the entire response to be loaded as a Buffer before I can write it to my server's response stream.

Answer 1 · 2012-10-30T20:44:13.000Z

Yes, you're right. It's something I've wanted to add for a while.

There is a pull request I need to review and update so that it fits a little bit more nicely with the way AwsSum works.

https://github.com/appsattic/node-awssum/pull/54

I should get to it pretty soon.

Cheers,
Andy

Answer 2 · 2012-12-04T12:30:58.000Z

FWIW, this is the only gripe we have with using awssum as our main tool for handling S3 uploads/downloads. Holding an entire body in-memory isn't really feasible as it slurps up the appserver's available memory pretty fast.
Having some kind of alias that allows you to upload a file (without knowing the filesize) would be very useful as well. I think this can be done by using the multipart APIs but that adds quite a bit of maintenance overhead on the implementer's side. It would be pretty awesome if awssum could abstract this away.

Answer 3 · 2012-12-04T18:27:07.000Z

Hi Simon,

Yep, you're right (in part).

If you want to stream from disk for a PutObject you can already do this. Take a look at the following page:

http://awssum.io/amazon/s3/put-object.html

There isn't a way to stream from S3 yet ... but I promise, I'm working on it. :)

Cheers,
Andy

Answer 4 · 2012-12-05T18:16:53.000Z

What is the status of this, streaming downloads would be very nice?

Answer 5 · 2012-12-05T18:29:37.000Z

Hi Guys,

I was working on it yesterday but I need to just do a slight re-arrangement to make it work (and understand a bit more about streams of course). I'll be able to have another go at it today. Fingers crossed it should be there in a day or two. :)

Cheers,
Andy

Answer 6 · 2012-12-05T19:55:27.000Z

Hi Guys,

I've pushed what I think is an initial go at streaming downloads. Have a look at the example here:

https://github.com/appsattic/node-awssum/blob/master/examples/amazon/s3/get-object-stream.js

Please let me know if anything looks weird. It'd be great if you could all test it since it'll definitely require some hard use cases.

Cheers,
Andy

Answer 7 · 2012-12-05T20:26:52.000Z

Minor update to make sure we call the callback with an error if the status code is wrong.

Answer 8 · 2013-02-07T12:56:14.000Z

Any plans to release it soon?

Answer 9 · 2013-02-07T18:28:24.000Z

Hiya,

Yes, I'm hoping to release in the next week or so. I'm afraid it's all dependent on how much time I have though. :)

Cheers,
Andy