S3 Publisher

Stargaze if you find it useful.

Provides infrastructure for uploading multiple artifacts to S3 bucket using signed url, that could be processed and uploaded to a defined target.

Overview

This Application is designed to leverage the Serverless architecture by using the AWS Lambda functions and HttpApi which uses an Authorizer for Client authentication.

The communication flow is based on a decoupled design so that the Lambda function for uploading the test artifacts is not directly used. Instead, after a successful client authentication, an S3 signed Url is generated that is used to upload the artifacts.

System Requirements

Docker
VS Code
AWS Cli (configured)

Already in Dev Container:

Python 3.9
NodeJs 16.x
AWS SAM
jq
pre-commit
black

AWS Serverless Application Model was used to design, build and deploy this application which uses Lambda function, HttpApi and S3 buckets.

SAM infrastructure as code template is inherited from CloudFormation and the current template is used to deploy all infrastructure used by this application.

Dev Container

After cloning the git repository, open the folder in VS Code and if docker is running, it asks you to open the workspace in Dev Container.

All required tools are already installed in the Dev Container, and you could start working.

Deployment

For deploying the application in a new environment, you should package and publish the application to sam s3 bucket once, after that all subsequent build and deployments could be applied using related sam commands.

Using deploy.sh script, it would validate the sam template, build and deploy it to the configured aws region. It also adds the missing lambda notification.

After deployment, please configure the environment variables for the authorizer and publish artifacts lambda functions.

Test

For testing the functionality, switch to the test folder and run upload-log.py command, it will get the api endpoint url, pack both log files, then authenticate the client and get the signed url for uploading the packed file to s3 bucket.

The idea of this solution is, since S3 supports only a single file per upload, we pack our files, in this example 2 files as tar.gz and upload them to S3 where it will be unpacked and processed by publishArtifacts lambda function.

Please update the authorization environment variable according to the environment value you set in the authorizer function. For more information, look into the AuthorizerAPIKey variable in the sam template.

/workspaces/s3-publisher/app/test (main) $ python3 upload-log.py test-log1.xml test-log1.json

Local testing keyAuthorizer

/workspaces/s3-publisher (main) $ sam local invoke ApiKeyAuthorizer -e keyAuthorizer/requestEvent.json -n keyAuthorizer/env.json

Local testing getSignedURL

/workspaces/s3-publisher (main) $ sam local invoke UploadRequest

Cleanup

In order to remove all created aws resources during deployment, run the following command in app folder

/workspaces/s3-publisher (main) $ sam delete

Code formatting

Automatic Code formatting is done using pre-commit hooks.

pre-commit manages all of your hooks using a yaml config file: .pre-commit-config.yaml

When you run git commit command, it will first execute the configured pre-commit hooks for the staged files.

Code Analysis

Trivy

Trivy action is based on Trivy vulnerability scaner. Using GitHub Actions, following workflows are configured:

Scanning the Dev Container image
Scanning IaC template and Dockerfile
Scanning the repository

Local Infrastructure as Code analysis

vscode ➜ /workspaces/s3-publisher (main ✗) $ docker run -it --rm -v $PWD:/workspace -w /workspace aquasec/trivy:0.21.0 config --format json  --output trivy-results.json .

CodeQL

Using GitHub CodeQL Action, semantic code analysis for python and javascript is configured.

simorgh1/s3-publisher