This repo contains several utilities for Data Science and ML Engineering work. I made this both for personal use and for practice using AWS.
The aws-stacks
directory contains several CloudFormation stacks as .yml
files.
The idea is to have stacks for everyday tasks such as training an ML model, or setting
up a logging / tracking server.
The assets
directory contains bootstrap scripts and configuration files that may be used
with the stacks. The assets associated with a given stack, say my-stack.yml
, in aws-stacks
should be stored in a directory assets/my-stack
, that is, with the same name as the stack.
In order to add a new stack, first create the CloudFormation .yml
template and store it in
the aws-stacks
directory. When making the template, take the following into account:
- If the templates uses a CIDR range of addresses to allow SSH access
to an instance and / or the name of a key to gain SSH access, name these parameters
SSHAccessAddresses
andKeyNameParameter
, respectively. - If the templates uses assets that must be copied to the instance on startup (such as a bootstrap script),
save these files in the
assets
folder under a subdirectory with the same name as the stack, as was said above. Additionally, you should add a parameterAssetsBucketName
to the stack (no default value) to pass the name of the assets bucket, and your instances should have permissions to read from this bucket.
In order to set up the library for use, do the following:
- Create an S3 bucket to store the stacks and assets.
- Ensure you have the aws CLI installed.
- Run the
setup.sh
script withbash setup.sh your-bucket-name
. This will create a~/.ds-utils-data
directory to store metadata. Within it there will be a.env
file with the bucket name. Additionally, it will sync youraws-stacks
andassets
directories to the bucket.
If you have already performed the initial setup and have added new stacks or assets, you can once
again run the setup.sh
script with bash setup.sh
to sync these to the bucket.
In order to install the python library to manage the stacks, first navigate to the python-lib
directory
and then run the setup script with
python setup.py install
Once you have installed the python library, you can launch a stack more easily with
python -m aws_utils_lib.cf_stack launch --stack_name=a-stack-name
To add parameters, such as KeyNameParameter
, for example, you can add them as flags. So for example to set this
parameter you could run
python -m aws_utils_lib.cf_stack launch --stack_name=a-stack-name --KeyNameParameter=a-key-name
Additionally, you can also delete a stack with the python library as follows:
python -m aws_utils_lib.cf_stack delete --stack_name=a-stack-name
You can also use the library to list the currently deployed stacks with:
python -m aws_utils_lib.cf_stack active-stacks
The versioning of the python library should use these guidelines:
- Tiny version change (
0.0.X
): Minor change or bugfix. - Minor version change (
0.X.0
): New features or functionality. - Major version change (
X.0.0
): Change that breaks or substantially alters existing functionality or API.