Support in-memory data for Transfer Manager download_chunks_concurrently and upload_chunks_concurrently
MattIrv opened this issue · 1 comments
MattIrv commented
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
- Today Transfer Manager's
download_chunks_concurrently
andupload_chunks_concurrently
require you to provide a filename. Unfortunately I have bytes that are already in-memory and may not fit on disk (or it may be slow to persist it to disk) and can't use Transfer Manager as a result.
A clear and concise description of what you want to happen.
- Provide functions that support the same interface as
download_chunks_concurrently
andupload_chunks_concurrently
but take a file-like object instead of a filename - Or, for write, provide a similar function that returns a stream to the caller, and then writes to the stream are uploaded via multipart upload as needed (not sure if there's any realistic similar thing here for reads)
- It's ok if this functionality can only be threaded and not use multi-processing since in practice we still see a large speed increase from threading