/s3-kms

S3 KMS

Primary LanguageJavaScript

AWS S3 KMS Playground

Build Status

Download all objects in a given S3 bucket recursively and save them locally, maintain directory structure. Encrypt all downloaded objects using KMS with a user-defined CMK and upload them to a second S3 bucket.

Setting Up Local Codebase

Install packages

npm i

Run Linting

npm run lint
npm run lint:fix

Run Tests

npm run test
npm run test:integration

Local AWS Infrastructure Setup

  • Spins up local AWS infrastructure via localstack along with S3 + KMS bootstrap
  • Refer /localstack/scripts/setup.sh for bootstrap
  • Make sure docker is running on your local machine before spinning up localstack
npm run start:infra

Destroys the local AWS infrastructure

npm run destroy:infra

How to run

  • Download S3 files recursively & download to the local machine
npm run start:download
  • Upload files to KMS encrypted bucket
npm run start:upload

How to test

  • Copy .env.sample to .env for configuration
  • Recursively upload files from localstack/sample/* to S3
  • Copy downloaded files to .tmp/*
  • To test on AWS update the configuration file
  • To test on local machine, spin up local stack, follow infra commands above
  • Local infrastructure localstack/scripts/setup.sh comprises of the bootstrap script to run aws cli (Note: KMS is not supported in localstack)
  • Download & upload files has it's own script command

KMS with CMK

  • Create access policy to manage CMK & encrypt S3 objects
  • Create KMS key with the policy
    aws kms create-key --region us-west-2 --policy file://kms-policy.json
  • Configure the S3 bucket SSE
    aws --region=us-west-2 --endpoint-url=http://localstack:4572 s3api put-bucket-encryption --bucket encrypted-bucket --server-side-encryption-confi

Benchmark Tests

These tests have been ran and reported on Mac OSX v10.14.5 15 inch 2.8 GHz Intel Core i7

  • Copy files recursively on local machine // dir size: 62kb
====================== sortBy ======================
Slow  Promise.all            31 ms/op      3 samples
      Promise.map(3)         29.29 ms/op   3 samples
Fast  Promise.map(Infinity)  27.34 ms/op   3 samples
  • Upload files from local to S3 // dir size: 62kb
====================== sortBy ======================
Slow  Promise.map(Infinity)     29.06 ms/op   6 samples
      Promise.map(10)           28.40 ms/op   6 samples      
Fast  Promise.map(3)            26.91 ms/op   6 samples

TODO

  • Add prettier, eslint
  • Add jest for unit test
  • Add aws-sdk, bluebird
  • Setup AWS infrastructure (S3 + SSE KMS Encyption with CMK)
  • Setup localstack for local AWS environment
  • Copy files recursively on local machine from S3
  • Unit tests for copy files
  • Performance benchmark tests for copy
  • Upload the files to S3 using SSE with KMS
  • Unit test for upload files
  • Performance benchmark tests for upload
  • Integration tests