/Multi-threaded-downloader

A http file downloader made in nodejs

Primary LanguageJavaScriptMIT LicenseMIT

mt-downloader

Join the chat at https://gitter.im/tusharmath/Multi-threaded-downloader Build Status npm Commitizen friendly semantic-release js-standard-style Coverage Status

This is a nodejs based utility library that helps in performing resumable, multi-threaded downloads over Http. The module is highly inspired by Speedbit's — Download Accelerator Plus.

Features

  1. Multi connection downloads: In a conventional download we don't completely utilize our bandwidth. With multiple connections there is always one thread which is getting data thus minimizing the wait period between data packets. This doesn't mean that we will be able to download faster than what our ISP allows.

  2. Stop and start from the last downloaded byte: You don't have to worry about internet getting disconnected or your computer shutting down while downloading. You can quite easily start from the last byte that was downloaded.

  3. Console application: If installed globally, mtd command would be available.

Installation

The conventional npm installation process needs to be followed.

npm install mt-downloader --save

CLI Installation

$ npm install -g mt-downloader
$ mtd --help

Objects

Functions

FILE : object

Provides wrappers over the async utils inside the fs module. The wrappers take in an input stream of arguments and returns the result of function call as another stream.

Kind: global namespace

FILE.open(params$) ⇒ Observable

Kind: static method of FILE

Param Type
params$ Observable

FILE.fstat(params$) ⇒ Observable

https://nodejs.org/api/fs.html#fs_fs_open_path_flags_mode_callback

Kind: static method of FILE

Param Type
params$ Observable

FILE.read(params$) ⇒ Observable

https://nodejs.org/api/fs.html#fs_fs_read_fd_buffer_offset_length_position_callback

Kind: static method of FILE

Param Type
params$ Observable

FILE.write(params$) ⇒ Observable

https://nodejs.org/api/fs.html#fs_fs_write_fd_buffer_offset_length_position_callback

Kind: static method of FILE

Param Type
params$ Observable

FILE.close(params$) ⇒ Observable

https://nodejs.org/api/fs.html#fs_fs_close_fd_callback

Kind: static method of FILE

Param Type
params$ Observable

FILE.truncate(params$) ⇒ Observable

https://nodejs.org/api/fs.html#fs_fs_truncate_path_len_callback

Kind: static method of FILE

Param Type
params$ Observable

FILE.rename(params$) ⇒ Observable

https://nodejs.org/api/fs.html#fs_fs_rename_oldpath_newpath_callback

Kind: static method of FILE

Param Type
params$ Observable

HTTP : object

Kind: global namespace

HTTP.request(params) ⇒ Observable

Stream based wrapper over npm/request

Kind: static method of HTTP
Returns: Observable - multiplex stream

Param Type Description
params object request module params.

CreateMTDFile(options) ⇒ Observable

Creates a new .mtd file that is a little larger in size than the original file. The file is initially empty and has all the relevant meta information regarding the download appended to the end.

Kind: global function
Returns: Observable - A multiplexed stream containing ~

  • written$ - Bytes being saved on disk.
  • meta$ - Meta information about the download.
  • remoteFileSize$ - Size of the content that is to be downloaded.
  • fdW$ - File descriptor in w mode.
Param Type Default Description
options object The options must have mtdPath and url.
options.url string Download url.
options.path string Relative path where the file needs to be saved.
[options.range] number 3 Number of concurrent downloads.
[options.metaWrite] number 300 Throttles the write frequency of meta data.

DownloadFromMTDFile(mtdPath, [meta]) ⇒ Observable

Reads a .mtd file and resumes the download from the last successfully saved byte.

Kind: global function
Returns: Observable - A multiplexed stream containing ~

  • metaWritten$ - Meta data buffer stream.
  • response$ - HTTP response object.
  • responses$ - List of all the HTTP response objects.
  • localFileSize$ - Size of the .mtd file on disk.
  • fdR$ - File Descriptor in r+ mode.
  • meta$ - Download meta information.
Param Type Description
mtdPath String Relative path to the .mtd file.
[meta] Object Optional meta data to override the one that's being loaded from the .mtd file.

FinalizeDownload(params) ⇒ Observable

Removes the meta information and the .mtd extension from the file once the download is successfully completed.

Kind: global function
Returns: Observable - A multiplexed stream containing ~

  • truncated$ - Fired when the meta data is removed.
  • renamed$ - Fired when the .mtd extension is removed.
Param Type Description
params object {fd$, meta$}
params.fd$ Observable File descriptor Observable
params.meta$ Observable Download meta information

Completion(meta$) ⇒ Observable

Util method that calculates the total completion percentage (between 0-100).

Kind: global function
Returns: Observable - Value between 0-100

Param Type Description
meta$ Observable Meta data stream ie. exposed by DownloadFromMTDFile

.mtd file

Once the download starts the library will create a file with a .mtd extension. This file contains some meta information related to the download and is a little bigger (around 4kb) than the original file size. The .mtd file can be used later to restart downloads from where the last byte that was downloaded. After the download is completed the downloader will truncate the .mtd file to remove that meta information.

Releases

There are two release channels viz. — latest and next. The next is unstable and must be used with caution.

Latest

npm i mt-downloader

Next

npm i mt-downloader@next