I created this demo repo because documentation for multipart uploading of large files using presigned URLs was very scant.
I wanted to create a solution to allow users to upload files directly from the browser to AWS S3 (or any S3-compliant storage server). This worked great when I used AWS SDK's getSignedUrl API to generate a temporary URL that the browser could upload the file to.
However, I hit a snag when dealing with files > 5GB because the pre-signed URL only allows for a maximum file size of 5GB to be uploaded at one go. As such, this repo demonstrates the use of multipart + presigned URLs to upload large files to an AWS S3-compliant storage service.
- Frontend Server: React (Next.js)
- Backend Server: Node.js (Express), using the AWS JS SDK
- Storage Server: Minio (but this can easily be switched out to AWS S3)
- Clone the repo and change directory into the repo
- Open three different terminal windows.
Storage Server
In window 1, run:
# Set up the Minio server (ignore this if you are using AWS S3)
# Minio docs: https://docs.minio.io/docs/minio-quickstart-guide
minio server /data
Note: Set the S3-compliant bucket policy as appropriate to allow the right access
Backend Server
Replace the following code in backend/server.js with your AWS S3 or S3-compliant storage server config.
const s3 = new AWS.S3({
accessKeyId: '<ACCESS_KEY_ID>' , // Replace with your access key id
secretAccessKey: '<SECRET_ACCESS_KEY>' , // Replace with your secret access key
endpoint: 'http://127.0.0.1:9000' ,
s3ForcePathStyle: true, // needed with minio?
signatureVersion: 'v4'
});
Note: If you are using AWS S3, follow the docs on the AWS website to instantiate a new AWS S3 client.
In window 2, run:
cd backend
npm install
node server.js
Frontend Server
In window 3, run:
cd frontend
npm install
npm run dev
Upload File
Go to http://localhost:3000 in your browser window and upload a file.