/s3-upload-stream

Code library that provides a Stream implementation that makes working with uploads to S3 easier where the size of the content isn't known a priori. It holds only partial content in memory (works with large objects), and is compatible with code libraries that work with output streams.

Primary LanguageC#MIT LicenseMIT

S3 Upload Stream

This library performs uploads to S3 without holding the entire object content in memory (or storage) beforehand. This approach is more efficient than using MemoryStream and offers compatibility with libraries and packages that work with a Stream interface. Examples for extracting GZIP, decrypting AES and PGP, and uploading dynamically-generated CSV are provided.

This project is a follow-on to SeekableS3Stream, also on Github. Using both libraries allows for very efficient, and simple, large-file processing to and from S3 via AWS Lambda, as demonstrated in one of the Examples.

For a more detailed explaination, check out the article on Medium.