/s3-stream-commander

Multipart streaming download utility for S3

Primary LanguageJavaScriptMIT LicenseMIT

s3-stream-commander

Multipart streaming download utility for S3. Note: Forked from s3-stream-download, added bubbling up of errors and uses s3.headObject/getObject to be more performant.

Features

Installation

  $ npm install s3-stream-commander --save

Usage

var AWS = require('aws-sdk');
var S3StreamCommander = require('./index');
var fs = require('fs');

var s3 = new AWS.S3();

var s3StreamCommander = new S3StreamCommander(s3, {Bucket: '<BUCKET>', Key:'<KEY>'});

s3StreamCommander.pipe(fs.createWriteStream('<FILENAME>'));

s3StreamCommander.on('end', function() {
    process.exit(0);
});

Documentation

new S3StreamCommander(s3, s3Params, options)

Creates a new instance of a multipart download stream. This is a readable stream that can be piped into other streams.

Arguments

  • s3 - Configured aws-sdk s3 instance. Should be preconfigured with any credentials.
  • s3Params - Params object that would normally be passed to s3.getObject()
  • options - Options
    • downloadChunkSize - Size of each chunk in bytes. Defaults to 5MB.
    • concurrentChunks - Number of chunks to download concurrently. Defaults to 5.
    • retries - How many times a failed chunk should be retried before failing the entire download. Retries will exponentially backoff to allow for recovery. Defaults to 5.

People

The original author is Chris Kinsman

License

MIT