oyster-streamable

A proof of concept, streamable implementation of the Oyster Protocol (not fully compatible) with a few significant changes:

CryptoJS replaced with node-forge
Encryption/decryption key is the SHA-256 hash of the handle, per-chunk IV (see #109)
Communication with the nodes via IXI module to save bandwidth
Pipelined processing:
- File > Encryption > Upload
- Download > Decryption > Blob

API Reference

oyster-streamable

Importing oyster-streamable

Example (As an ES Module)

import Oyster from 'oyster-streamable'

Example (As a Node Module)

const Oyster = require(`oyster-streamable`)

Example (As a UMD Module)

window.Oyster = Oyster.default

oyster-streamable
- .Download
  - .EVENTS
  - .toBuffer(handle, options) ⇒ Download
  - .toBlob(handle, options) ⇒ Download
- .Upload
  - .EVENTS
  - .EVENTS
  - .fromFile(file, options) ⇒ Upload
  - .fromData(buffer, filename, options) ⇒ Upload

Oyster.Download

Kind: static class of oyster-streamable
Emits: METADATA, DOWNLOAD_PROGRESS, FINISH

.Download
- .EVENTS
- .toBuffer(handle, options) ⇒ Download
- .toBlob(handle, options) ⇒ Download

Download.EVENTS

Events fired during the download lifecycle

Kind: static constant of Download

.EVENTS

"DOWNLOAD_PROGRESS"

Fired when a successful poll is performed while retrieving a file

Kind: event emitted by EVENTS
Properties

Name	Type	Description
progress	`Object`	a progress object
progress.progress	`Number`	the percentage of progress for the download

"FINISH"

Fired when the file has been reconstructed and is ready for use

Kind: event emitted by EVENTS
Properties

Name	Type	Description
file	`File` \| `Buffer`	the file as an object as the target type of the download instance
metadata	`Object`	the metadata object associated with the file

"METADATA"

Fired when the file metadata has been reconstructed and is ready for use

Kind: event emitted by EVENTS
Properties

Name	Type	Description
fileName	`String`	the name of the file being downloaded
ext	`String`	the file extension of the file being downloaded
numberOfChunks	`Number`	the number of chunks that the file is stored in

Download.toBuffer(handle, options) ⇒ `Download`

Kind: static method of Download

Param	Type	Default	Description
handle	`String`		the handle of the file to download
options	`Object`		the options for the download
options.iotaProviders	`Array.<Object>` \| `Array.<IOTA>`		an array of IOTA initialization Objects or IOTA instances
[options.autoStart]	`Boolean`	`true`	immediately start the download

Example (To Buffer object (node))

const download = Oyster.Download.toBuffer(handle, {
  iotaProviders: [
    { provider: 'https://poll.oysternodes.com:14265/' },
    { provider: 'https://download.oysternodes.com:14265/' }
  ]
})

download.on('meta', metadata => {
  console.log(metadata)
  // {fileName: "oyster.txt", ext: "txt", numberOfChunks: 2}
})
download.on('finish', filedata => {
  console.log(filedata)
  // {file: Buffer(), metadata: {…}, target: Download}
})

Download.toBlob(handle, options) ⇒ `Download`

Kind: static method of Download

Param	Type	Default	Description
handle	`String`		the handle of the file to download
options	`Object`		the options for the download
options.iotaProviders	`Array.<Object>` \| `Array.<IOTA>`		an array of IOTA initialization Objects or IOTA instances
[options.autoStart]	`Boolean`	`true`	immediately start the download

Example (To Blob object (browser))

const download = Oyster.Download.toBlob(handle, {
  iotaProviders: [
    { provider: 'https://poll.oysternodes.com:14265/' },
    { provider: 'https://download.oysternodes.com:14265/' }
  ]
})

download.on('meta', metadata => {
  console.log(metadata)
  // {fileName: "oyster.txt", ext: "txt", numberOfChunks: 2}
})
download.on('finish', filedata => {
  console.log(filedata)
  // {file: Blob(), metadata: {…}, target: Download}
})

Oyster.Upload

Kind: static class of oyster-streamable
Emits: INVOICE, CHUNKS_PROGRESS, UPLOAD_PROGRESS, FINISH

.Upload
- .EVENTS
- .EVENTS
- .fromFile(file, options) ⇒ Upload
- .fromData(buffer, filename, options) ⇒ Upload

Upload.EVENTS

Events fired during the upload lifecycle

Kind: static constant of Upload

.EVENTS

"INVOICE"

Fired when an invoice is recieved from the broker node

Kind: event emitted by EVENTS
Properties

Name	Type	Description
handle	`String`	the handle of the file uploaded
address	`String`	an ethereum address to send the pearl to
cost	`Number`	the cost of the file upload

"CHUNKS_PROGRESS"

Fired when a chunk is uploaded to the broker

Kind: event emitted by EVENTS
Properties

Name	Type	Description
progress	`Object`	a progress object
progress.progress	`Number`	the percentage of progress for the chunk upload

"UPLOADED"

Fired all chunks have been uploaded to the brokers.

Kind: event emitted by EVENTS

"META_ATTACHED"

Fired when the meta chunk has been attached. This is needed in order to resume polling for upload progress.

Kind: event emitted by EVENTS
Properties

Name	Type	Description
target	`Object`	the upload object
handle	`String`	the handle of the uploaded file
numberOfChunks	`Number`	the number of chunks for the file
metadata	`Object`	the metadata object

"UPLOAD_PROGRESS"

Fired when a chunk is attached to the tangle

Kind: event emitted by EVENTS
Properties

Name	Type	Description
progress	`Object`	a progress object
progress.progress	`Number`	the percentage of progress for the chunk attachment

"FINISH"

Fired when the file has been completely attached to the tangle

Kind: event emitted by EVENTS
Properties

Name	Type	Description
handle	`String`	the handle of the file uploaded
metadata	`Object`	the metadata object associated with the file

Upload.EVENTS

Events fired during the upload lifecycle

Kind: static constant of Upload

.EVENTS

"INVOICE"

Fired when an invoice is recieved from the broker node

Kind: event emitted by EVENTS
Properties

Name	Type	Description
handle	`String`	the handle of the file uploaded
address	`String`	an ethereum address to send the pearl to
cost	`Number`	the cost of the file upload

"CHUNKS_PROGRESS"

Fired when a chunk is uploaded to the broker

Kind: event emitted by EVENTS
Properties

Name	Type	Description
progress	`Object`	a progress object
progress.progress	`Number`	the percentage of progress for the chunk upload

"UPLOADED"

Fired all chunks have been uploaded to the brokers.

Kind: event emitted by EVENTS

"META_ATTACHED"

Fired when the meta chunk has been attached. This is needed in order to resume polling for upload progress.

Kind: event emitted by EVENTS
Properties

Name	Type	Description
target	`Object`	the upload object
handle	`String`	the handle of the uploaded file
numberOfChunks	`Number`	the number of chunks for the file
metadata	`Object`	the metadata object

"UPLOAD_PROGRESS"

Fired when a chunk is attached to the tangle

Kind: event emitted by EVENTS
Properties

Name	Type	Description
progress	`Object`	a progress object
progress.progress	`Number`	the percentage of progress for the chunk attachment

"FINISH"

Fired when the file has been completely attached to the tangle

Kind: event emitted by EVENTS
Properties

Name	Type	Description
handle	`String`	the handle of the file uploaded
metadata	`Object`	the metadata object associated with the file

Upload.fromFile(file, options) ⇒ `Upload`

Kind: static method of Upload

Param	Type	Default	Description
file	`File`		the file to upload
options	`Object`		the options for the upload
options.iotaProvider	`Object` \| `IOTA`		an IOTA initialization Object or IOTA instance
options.alpha	`String`		the endpoint for the alpha broker
options.beta	`String`		the endpoint for the beta broker
options.epochs	`Number`		the number of years to store the file
[options.autoStart]	`Boolean`	`true`	immediately start the upload

Example (From File object (browser))

const file = fileInput.files[0];
const upload = Oyster.Upload.fromFile(file, {
  iotaProvider: { provider: 'https://poll.oysternodes.com:14265/' },
  alpha: 'https://broker-1.oysternodes.com/',
  beta: 'https://broker-2.oysternodes.com/',
  epochs: 1
});

upload.on('invoice', invoice => {
  console.log(invoice)
  // {address: "<ETH_ADDRESS>", cost: 20}
});
upload.on('finish', filedata => {
  console.log(filedata)
  // {handle: "<OYSTER_HANDLE>", metadata: {…}, target: Upload}
});

Upload.fromData(buffer, filename, options) ⇒ `Upload`

Kind: static method of Upload

Param	Type	Default	Description
buffer	`Buffer`		the data Buffer to upload
filename	`String`		the name of the file
options	`Object`		the options for the upload
options.iotaProvider	`Object` \| `IOTA`		an IOTA initialization Object or IOTA instance
options.alpha	`String`		the endpoint for the alpha broker
options.beta	`String`		the endpoint for the beta broker
options.epochs	`Number`		the number of years to store the file
[options.autoStart]	`Boolean`	`true`	immediately start the upload

Example (From Buffer object (node))

const fs = require('fs');
const path = './path/to/file';
const filename = 'somefile.txt';

fs.readFile(`${path}/${filename}`, (err, data) => {
  if (err) throw err;

  const upload = Oyster.Upload.fromData(data, filename, {
    iotaProvider: { provider: 'https://poll.oysternodes.com:14265/' },
    alpha: 'https://broker-1.oysternodes.com/',
    beta: 'https://broker-2.oysternodes.com/',
    epochs: 1
  });

  upload.on('invoice', invoice => {
    console.log(invoice)
    // {address: "<ETH_ADDRESS>", cost: 20}
  });
  upload.on('finish', filedata => {
    console.log(filedata)
    // {handle: "<OYSTER_HANDLE>", metadata: {…}, target: Upload}
  });
});

What's next?

Possible things to look into:

Full compatibility
Pluggable source streams for uploads and target streams for downloads, allowing:
Node compatability
Video streams via MediaSource
Arbitrary size downloads via StreamSaver

Credits

This repo is based on the official oysterprotocol/webinterface and shares a fair amount of code with the project where it makes sense.

Big thanks to everyone working on Oyster!

oysterprotocol/oyster-streamable

oyster-streamable

API Reference

oyster-streamable

Oyster.Download

Download.EVENTS

"DOWNLOAD_PROGRESS"

"FINISH"

"METADATA"

Download.toBuffer(handle, options) ⇒ Download

Download.toBlob(handle, options) ⇒ Download

Oyster.Upload

Upload.EVENTS

"INVOICE"

"CHUNKS_PROGRESS"

"UPLOADED"

"META_ATTACHED"

"UPLOAD_PROGRESS"

"FINISH"

Upload.EVENTS

"INVOICE"

"CHUNKS_PROGRESS"

"UPLOADED"

"META_ATTACHED"

"UPLOAD_PROGRESS"

"FINISH"

Upload.fromFile(file, options) ⇒ Upload

Upload.fromData(buffer, filename, options) ⇒ Upload

What's next?

Credits

Download.toBuffer(handle, options) ⇒ `Download`

Download.toBlob(handle, options) ⇒ `Download`

Upload.fromFile(file, options) ⇒ `Upload`

Upload.fromData(buffer, filename, options) ⇒ `Upload`