Instructions

Basic deid usage is also in the Makefile

# Will run deid starting from the bucket root
make deid_ohsu
make deid_washu

# Example "make deid_ohsu"
python scripts/aws_detect_pii.py \
    -b htan-dcc-ohsu \
    --bucket-type aws \
    --profile sandbox-developer \
    --comprehend_profile htan-dev-admin \
    > outputs/test_ohsu_output.tsv

python scripts/aws_detect_pii.py \
    -b htan-dcc-washu \
    --bucket-type gcs \
    --profile htan-gcs \
    --comprehend_profile htan-dev-admin \
    > outputs/test_washu_output.tsv

Notes

Deid starting at the bucket root takes a long time. Suggest you supply the optional --prefix tag to the aws_detect_pii.py script.

python scripts/aws_detect_pii.py \
    -b htan-dcc-ohsu \
    --prefix imaging_level_2 \
    --bucket-type aws \
    --profile sandbox-developer \
    --comprehend_profile htan-dev-admin \
    > outputs/test_ohsu_output.tsv

lambda-template

A GitHub template for quickly starting a new AWS lambda project.

Naming

Naming conventions:

for a vanilla Lambda: lambda-<context>
for a Cloudformation Transform macro: cfn-macro-<context>
for a Cloudformation Custom Resource: cfn-cr-<context>

Development

Contributions

Contributions are welcome.

Requirements

Run pipenv install --dev to install both production and development requirements, and pipenv shell to activate the virtual environment. For more information see the pipenv docs.

After activating the virtual environment, run pre-commit install to install the pre-commit git hook.

Create a local build

$ sam build

Run unit tests

Tests are defined in the tests folder in this project. Use PIP to install the pytest and run unit tests.

$ python -m pytest tests/ -v

Run integration tests

Running integration tests requires docker

$ sam local invoke HelloWorldFunction --event events/event.json

Deployment

Deploy Lambda to S3

Deployments are sent to the Sage cloudformation repository which requires permissions to upload to Sage bootstrap-awss3cloudformationbucket-19qromfd235z9 and essentials-awss3lambdaartifactsbucket-x29ftznj6pqw buckets.

sam package --template-file .aws-sam/build/template.yaml \
  --s3-bucket essentials-awss3lambdaartifactsbucket-x29ftznj6pqw \
  --output-template-file .aws-sam/build/lambda-template.yaml

aws s3 cp .aws-sam/build/lambda-template.yaml s3://bootstrap-awss3cloudformationbucket-19qromfd235z9/lambda-template/master/

Publish Lambda

Private access

Publishing the lambda makes it available in your AWS account. It will be accessible in the serverless application repository.

sam publish --template .aws-sam/build/lambda-template.yaml

Public access

Making the lambda publicly accessible makes it available in the global AWS serverless application repository

aws serverlessrepo put-application-policy \
  --application-id <lambda ARN> \
  --statements Principals=*,Actions=Deploy

Install Lambda into AWS

Sceptre

Create the following sceptre file config/prod/lambda-template.yaml

template_path: "remote/lambda-template.yaml"
stack_name: "lambda-template"
stack_tags:
  Department: "Platform"
  Project: "Infrastructure"
  OwnerEmail: "it@sagebase.org"
hooks:
  before_launch:
    - !cmd "curl https://bootstrap-awss3cloudformationbucket-19qromfd235z9.s3.amazonaws.com/lambda-template/master/lambda-template.yaml --create-dirs -o templates/remote/lambda-template.yaml"

Install the lambda using sceptre:

sceptre --var "profile=my-profile" --var "region=us-east-1" launch prod/lambda-template.yaml

AWS Console

Steps to deploy from AWS console.

Login to AWS
Access the serverless application repository -> Available Applications
Select application to install
Enter Application settings
Click Deploy

Releasing

We have setup our CI to automate a releases. To kick off the process just create a tag (i.e 0.0.1) and push to the repo. The tag must be the same number as the current version in template.yaml. Our CI will do the work of deploying and publishing the lambda.

ncihtan/s3batch-phi-detector