aws-beam/aws-elixir

Can I stream a large object from S3?

Closed this issue · 2 comments

ryanwinchester commented

Is there a way to do this? I'm not seeing anything in the docs, so just wondering if it is possible.

philss commented

@ryanwinchester I think it's possible, but it would require to keep track of each section. There is a range parameter that you can use in https://hexdocs.pm/aws/AWS.S3.html#get_object/22, which makes this possible. I didn't play with it, but I will try and get back to you.

Also I think that it's possible to add more docs related to the parameters, because here you can find a more detailed explanation: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html

  • 👍1
philss commented

@ryanwinchester I made this small PoC to demonstrate how this would work using Stream.resource/3: https://github.com/philss/aws-s3-stream-download-poc/blob/main/lib/download_manager.ex#L22

I'm closing the issue. Feel free to reopen if you need to :)

  • 👍1
  • ❤️3