An outline for a proposed AWS pipeline. The pipeline is meant to take app zips and automatically deploy them as AWS Lambda functions.
That was the initial solution, and eventually the adopted solution, however it originally was scrapped due to design miscommunication.
- became familiar with the AWS ecosystem/tools
- learned the importance of effective communication and
- recognized the power of good/simple visualizations
This information may be useful in the future; why throw away hard work when you can quickly outline/document it and archive it.
- The zipped app code is uploaded to an S3 bucket.
- Upon upload completion, an S3 event is triggered resulting in an auto- generated message. This message contains the zip path, among other things.
- The message is placed onto an SQS message queue.
- The message is then picked off the queue by a worker inside of an auto- scaling group.
- Workers in this group run a custom AMI which uses a combination of the Amazon linux OS(in order to build native dependencies), bash scripts and chalice (or zappa) to create and deploy a lambda function for the app zip.
We want to use the queue size as our target scaling metric for the worker group (i.e. App Lambda Builder Group).
By default, cloudwatch tracks the queue size as a metric which is updated every 5 minutes. In order to increase the update frequency, we need to create a custom cloudwatch metric which uses a checker instance as its source (i.e. a small ec2 whose sole purpose is to poll the queue size at some specified frequency); this is the official AWS solution as outlined in Rapid Auto Scaling with Amazon SQS.