tusharmath/Multi-threaded-downloader

A http file downloader made in nodejs

mt-downloader

This is a nodejs based utility library that helps in performing resumable, multi-threaded downloads over Http. The module is highly inspired by Speedbit's — Download Accelerator Plus.

Features

0. Multi connection downloads: In a conventional download we don't completely utilize our bandwidth. With multiple connections there is always one thread which is getting data thus minimizing the wait period between data packets. This doesn't mean that we will be able to download faster than what our ISP allows.

1. Stop and start from the last downloaded byte: You don't have to worry about internet getting disconnected or your computer shutting down while downloading. You can quite easily start from the last byte that was downloaded.

2. Console application: If installed globally, mtd command would be available.

Installation

The conventional npm installation process needs to be followed.

npm install mt-downloader --save

CLI Installation

$ npm install -g mt-downloader
$ mtd --help

Objects

• FILE : object
  • .open(params$) ⇒ Observable
  • .fstat(params$) ⇒ Observable
  • .read(params$) ⇒ Observable
  • .write(params$) ⇒ Observable
  • .close(params$) ⇒ Observable
  • .truncate(params$) ⇒ Observable
  • .rename(params$) ⇒ Observable
• HTTP : object
  • .request(params) ⇒ Observable

Functions

• CreateMTDFile(options) ⇒ Observable
• DownloadFromMTDFile(mtdPath, [meta]) ⇒ Observable
• FinalizeDownload(params) ⇒ Observable
• Completion(meta$) ⇒ Observable

FILE : object

Provides wrappers over the async utils inside the fs module. The wrappers take in an input stream of arguments and returns the result of function call as another stream.

Kind: global namespace

• FILE : object
  • .open(params$) ⇒ Observable
  • .fstat(params$) ⇒ Observable
  • .read(params$) ⇒ Observable
  • .write(params$) ⇒ Observable
  • .close(params$) ⇒ Observable
  • .truncate(params$) ⇒ Observable
  • .rename(params$) ⇒ Observable

FILE.open(params$) ⇒ Observable

Kind: static method of FILE

Param	Type
params$	Observable

FILE.fstat(params$) ⇒ Observable

https://nodejs.org/api/fs.html#fs_fs_open_path_flags_mode_callback

Kind: static method of FILE

Param	Type
params$	Observable

FILE.read(params$) ⇒ Observable

https://nodejs.org/api/fs.html#fs_fs_read_fd_buffer_offset_length_position_callback

Kind: static method of FILE

Param	Type
params$	Observable

FILE.write(params$) ⇒ Observable

https://nodejs.org/api/fs.html#fs_fs_write_fd_buffer_offset_length_position_callback

Kind: static method of FILE

Param	Type
params$	Observable

FILE.close(params$) ⇒ Observable

https://nodejs.org/api/fs.html#fs_fs_close_fd_callback

Kind: static method of FILE

Param	Type
params$	Observable

FILE.truncate(params$) ⇒ Observable

https://nodejs.org/api/fs.html#fs_fs_truncate_path_len_callback

Kind: static method of FILE

Param	Type
params$	Observable

FILE.rename(params$) ⇒ Observable

https://nodejs.org/api/fs.html#fs_fs_rename_oldpath_newpath_callback

Kind: static method of FILE

Param	Type
params$	Observable

HTTP : object

Kind: global namespace

HTTP.request(params) ⇒ Observable

Stream based wrapper over npm/request

Kind: static method of HTTP
Returns: Observable - multiplex stream

Param	Type	Description
params	object	request module params.

CreateMTDFile(options) ⇒ Observable

Creates a new .mtd file that is a little larger in size than the original file. The file is initially empty and has all the relevant meta information regarding the download appended to the end.

Kind: global function
Returns: Observable - A multiplexed stream containing ~

• written$ - Bytes being saved on disk.
• meta$ - Meta information about the download.
• remoteFileSize$ - Size of the content that is to be downloaded.
• fdW$ - File descriptor in w mode.

Param	Type	Default	Description
options	object		The options must have mtdPath and url.
options.url	string		Download url.
options.path	string		Relative path where the file needs to be saved.
[options.range]	number	3	Number of concurrent downloads.
[options.metaWrite]	number	300	Throttles the write frequency of meta data.

DownloadFromMTDFile(mtdPath, [meta]) ⇒ Observable

Reads a .mtd file and resumes the download from the last successfully saved byte.

Kind: global function
Returns: Observable - A multiplexed stream containing ~

• metaWritten$ - Meta data buffer stream.
• response$ - HTTP response object.
• responses$ - List of all the HTTP response objects.
• localFileSize$ - Size of the .mtd file on disk.
• fdR$ - File Descriptor in r+ mode.
• meta$ - Download meta information.

Param	Type	Description
mtdPath	String	Relative path to the .mtd file.
[meta]	Object	Optional meta data to override the one that's being loaded from the .mtd file.

FinalizeDownload(params) ⇒ Observable

Removes the meta information and the .mtd extension from the file once the download is successfully completed.

Kind: global function
Returns: Observable - A multiplexed stream containing ~

• truncated$ - Fired when the meta data is removed.
• renamed$ - Fired when the .mtd extension is removed.

Param	Type	Description
params	object	{fd$, meta$}
params.fd$	Observable	File descriptor Observable
params.meta$	Observable	Download meta information

Completion(meta$) ⇒ Observable

Util method that calculates the total completion percentage (between 0-100).

Kind: global function
Returns: Observable - Value between 0-100

Param	Type	Description
meta$	Observable	Meta data stream ie. exposed by DownloadFromMTDFile

.mtd file

Once the download starts the library will create a file with a .mtd extension. This file contains some meta information related to the download and is a little bigger (around 4kb) than the original file size. The .mtd file can be used later to restart downloads from where the last byte that was downloaded. After the download is completed the downloader will truncate the .mtd file to remove that meta information.

Releases

There are two release channels viz. — latest and next. The next is unstable and must be used with caution.

Latest

npm i mt-downloader

Next

npm i mt-downloader@next