Is it possible to stream directly to S3 without uploading to `uploadDir` first?

[al6x]

Hello, is it possible to stream files directly to any write buffer (AWS S3 for example) without uploading to temporary uploadDir first?

[felixge]

No. The S3 API requires you to provide the size of new files when creating them. This information is not available for multipart/form-data files until they have been fully received. This means streaming is impossible.

[al6x]

Thanks, didn't know about such requirement from S3.

[mogadanez]

This is in case of PutObject

But we can use MultipartUpload:

//pseudocode:

File.prototype.open  =>  
this.uploadId = s3.createMultipartUpload (....)
this.partNumber= 0

File.prototype.write  => 
s3.uploadPart({partNumber:this.partNumber++, Body: buffer, UploadId : this.uploadId  })

File.prototype.open  =>  
s3.CompleteMultipartUpload(....)

[mogadanez]

In case of site with huge file upload temp-files can by an issue, because limited local drives.
For sure we have an options such as s3fs, to transparently mount uploadDir to Amazon. But i'm not sure about durability of this solution.

[Alexsey]

It's possible (may be some one else will need this)

In the code above I use bluebird to promisify the AWS api. You may do it with any other Promise library or just use it 'as-is' - callback based

// Top of your file
const Transform = require('stream').Transform

const AWS = require('aws-sdk')
// there are 4 ways to authenticate on S3. I use the .json file. You may choose
// http://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-credentials-node.html
AWS.config.loadFromPath('PASS_TO_YOUR_JSON_WITH_CREDENTIALS')
const formidable = require('formidable')
const s3 = new AWS.S3({signatureVersion: 'v4'})


// code above is in your route
const form = new formidable.IncomingForm()

form.parse(req) // <- node server request object in here
form.on('fileBegin', (name, file) => {
  file.on('error', e => this._error(e))

  file.open = function () {
    this._writeStream =  new Transform({
      transform (chunk, encoding, callback) {callback(null, chunk)}
    })

    this._writeStream.on('error', e => this.emit('error', e))

    s3.upload({
      Bucket: 'YOUR_EXISTING_BUCKET_NAME',
      Key: 'NEW_FILE_NAME',
      Body: this._writeStream
    }, onUpload)
  }

  file.end = function (cb) {
    this._writeStream.on('finish', () => {
        this.emit('end'))
        cb()
    }
    this._writeStream.end()
  }
})

// continue execution in here
function onUpload (err, res) {
  err ? console.log('error:\n', err) : console.log('response:\n', res)
}