file-stream

A NodeJS REST API HTTP server that supports the following:

Retrieves a list of files from AWS S3 bucket
Retrieves a specific file based on an appropriate unique identifier
Upload a single file into S3 bucket
Remove specific file from S3 bucket

Technical Overview

For this REST API server, NodeJS was used since it handles asynchronous IO very well especially when we read/write files either to the local directory or a remote server such AWS S3. Moreover, this implementation makes use of pipes and streaming so that buffered data will be sent/received in byte chunks instead of waiting for the whole file to be loaded in memory. The module FileUtil manages:

file encryption/decryption for security when we upload/download files
file compression/extraction to save on storage
file streaming in byte chunks for optimization and better user experience
option to read from / write to local directory The module AwsUtil manages:
performing read/write operations on AWS S3

Prerequisites

You need to have an AWS S3 account together with the corresponding access key id and secret access key

Usage

How this works:

Set various configurations (eg. server port, AWS access keys, etc.), you can edit the config file parameters in the config directory
Navigate to the root project directory then run npm install to install the npm modules
Start the REST API HTTP server by typing this on the command line terminal: node server.js (NOTE: you can also use pm2 clustering)

Tool and Testing

To access and use the tool to make various REST calls, enter this URL in the web browser:

http://127.0.0.1:3001/view/fileClient.html

To run the unit tests for the REST API calls, enter this in the command line terminal: npm test

REST API Endpoints

Once the server is up and running, you can make HTTP requests to these end points

Upload a file into the AWS S3 bucket
POST request parameters:
name: the unique identifier of the file to be uploaded

http://127.0.0.1:3001/file/:name

Retrieve a file from the AWS S3 bucket based on the specified identifier
GET request parameters:
name: the unique identifier of the file to be uploaded
extType: the extension / mime type of the file (eg. txt, png, mp4, etc.)

http://127.0.0.1:3001/file/:extType/:name

Delete a file from the AWS S3 bucket based on the specified identifier
DELETE request parameters:
name: the unique identifier of the file to be uploaded
extType: the extension / mime type of the file (eg. txt, png, mp4, etc.)

http://127.0.0.1:3001/file/:extType/:name

Retrieve the list of files from the AWS S3 bucket
GET request

http://127.0.0.1:3001/file/list

Dependencies

The code dependencies can be found in the npm modules list in this file package.json

License

This is an open source project. For more information, please refer to LICENSE

Peanoquio/file-stream