In this repository you will find the code to deploy all the resources to enforce controls on Glue Interactive Sessions.
Glue Interactive Sessions allows engineers to build, test and run data preparation and analytics workload in an interactive notebook. Interactive sessions provide isolated development environments, takes care of the underlying compute cluster and allows for configuration to stop idling resources. While interactive sessions come with default configuration, engineers can override options to customise the session to their needs. For example, provisioning more workers to experiment on a larger dataset or setting the idle timeout for long running workloads.
The controls defined here will terminate all the AWS Glue interactive based on the configuration to enforce Glue connections in vpc, maximum amount of workers and maixmum idle timeout. In addition will send an email to a user's specified address to alert regarding the termination.
The proposed architecture is built on serverless components and is executed event-driven whenever a new glue interactive session is created.
- Data engineer creates a new Glue Interactive Session either through the AWS Management Console or in a Jupyter Notebook locally.
- Glue Interactive Session will produce a new event toAWS CloudTrail for the CreateSession event with all relevant information to identify and inspect a session as soon as the session is initiated.
- An Amazon EventBridge rule filters the CloudTrail events and invokes the AWS Lambda function to inspect the CreateSession event.
- The Interactive Session Inspection function will inspect the CreateSession event and will check for all defined boundary conditions. Currently, the solution supports configuration for the maximum number of workers, idle timeout in minutes and deployment with connection enforced.
- If any of the defined boundary conditions are not met, for example too many workers are provisioned for the session, depending on the provided configuration the function will terminate the Glue Interactive Session immediately, will send an email via Amazon Simple Notification Service or both. In case the session is not started yet, the function will wait for it to start before terminating.
- In case the session was terminated, an e-mail is sent to an Amazon Simple Notification Service topic. There is no information available in the Glue Interactive Session notebook on the reason for the termination of the session. Therefore, through the SNS topic additional context information is provided to the data engineers.
- In case the function fails, the sessions are logged in a dead letter queue inside [Amazon Simple Queue]Service (https://aws.amazon.com/sqs/). Furthermore, the queue is monitored and in case of a message it will trigger an Amazon CloudWatch alarm.
cfn/template.yaml # Cloud Formation Template
src/functions # Lambda function and its library
test/ # unittest for lambda function
README.md # this readme file
requirements.txt # list of python library used from the solution and its tests
Makefile # List of tasks to conveniently install, build, deploy and delete the solution.All the necessary resources are defined in one AWS CloudFormation file located under cfn/template.yaml.
To deploy those resources we will use AWS Serverless Application Model (SAM), which will enable us to conventiently build and package all the dependencies and will also manage CloudFormation steps for us.
Following the main resources deployed by the CloudFormation Stack:
- One AWS Lambda function with its library, both defined under the directory
src/functions. The function is the control. It will validate that the session is started in a VPC and in case will terminate it. - One Amanzon Event Rule. This event will listen to AWS CloudTrail and in case of a new glue interactive session started will trigger the control lambda.
- One Amazon SQS Dead Letter Queue (DLQ) attached to the lambda function. Will keep record of events that triggered a lambda failure
- Two Amazon Cloud Watch Alarms monitoring the lambda failures and the messages in the DLQ.
Optionally 2 more resources will be deployed:
- One SNS topic
- One SNS Email subscription
In addition to previously defined resource, CloudFormation will also deploy all the necessary roles and policies plus one AWS KMS key to ensure that the exchanged data is encrypted.
To facilitate deployment lifecycle, included the setup of the user local environment, we are providing a Makefile where are described all the necessary steps.
- Ensure to have AWS credential renewed and access to your account.
How to do that is documented here - Explore the
Makefileand adapt to your needs the AWS Region and the Stack name by modifying the variablesAWS_REGIONandSTACK_NAME. - Set a value for
KILL_SESSION = "True"if you want to terminate immediately the Glue Interactive Session which has been found out of boundaries.
Allowed values:"True"||"False", default to"True" - Set a value for
NOTIFICATION_EMAIL_ADDRESS = <your.email@provider.com>in theMakefileif you want get notified when a session has been found out of boundaries. - Install all the prerequisites libraries:
These will be installed under a newly created python virtual environment inside this repository in the directory
make install-pre-requisites
.venv - Deploy the new stack
This command will:
make deploy
- check if the pre-requisites are satisfied
- perform pytest unittest on the python files
- validate the CloudFormation template
- build the artefacts (Lambda)
- deploy the resource via SAM.
Clean up the deployed resource by running when finished with your tests
make clean-upIn case the python3.9 is not an option and the previous commands are exiting with error mentioning python3.9 is missing, one solution would be to enable sam buildby using Docker.
As an example, you can change the line:
# sam build -t cfn/template.yaml ->
sam build --use-container -t cfn/template.yamlFor more information regarding sam build with docker please follow the official documentation here
This library is licensed under the MIT-0 License. See the LICENSE file.