This is a sample application that demonstrates how to build a sustainable architecture pattern according to the AWS Well-Architected Sustainability Pillar using the following services:
- Amazon Simple Queue Service(SQS)
- AWS Graviton Processors
- Amazon EC2 Spot Instances
- Amazon Simple Storage Service(S3)
- AWS Auto Scaling
- Amazon CloudWatch
Asynchronous and scheduled processing are key techniques for improving the sustainability of cloud architectures. In this lab, we will explore strategies for designing and implementing asynchronous, queue-driven systems on AWS to minimize resource usage and maximize efficiency.
This demo focuses specifically on the Software and architecture best practices for sustainability in the cloud.
This lab will use:
- Amazon SQS for queue-driven approach
- Auto-scaling workers adjusting dynamically to optimize resource utilization
- AWS Graviton CPUs for excellent energy efficiency
- Spot instances for reducing idle capacity
These are the steps of the workflow:
- We upload a picture to an S3 bucket
- The bucket is configured to trigger an event as soon as it receives a new file
- The event will be sent to an SQS queue
- As events gather, the queue will generate Amazon CloudWatch metrics
- Our Auto Scaling Group uses a Dynamic Scaling Policy based on a certain metric that will increase and decrease the number of EC2 Instances
- The Graviton & Spot based instances will retrieve the workload information from the queue and will execute it
- Original image is fetched and the processed one is uploaded into the S3 bucket
An automated deployment solution is provided with this demo using a CloudFormation template that you can provision directly in your own AWS account and in your Region of preference. The sqs-ec2-graviton-spot.yaml is used to set up all resources using AWS CloudFormation.
You can deploy the resources using the default parameters or modify them according to the description.
The stack includes an Amazon Simple Storage Service (S3) bucket with a CloudWatch trigger to push notifications to an SQS queue after an image is uploaded to the Amazon S3 bucket. The scope is to convert the jpg
image to pdf
. Once the message is in the queue, it is picked up by the application running on the EC2 instances in the Auto Scaling group. The EC2 instances are using sustainable and efficient Graviton processors. Moreover, this setup will leverage EC2 Spot instances. The image is converted to pdf
, and the instance is protected from scale-in for as long as it has an active processing job. After the workload has been successful processed, the Auto Scaling group will reduce the number of workers, thus reducing energy waste, lower costs, and minimize environmental footprints.
We will use the ApproximateNumberOfMessagesVisible
CloudWatch metric of the AWS/SQS
namespace to take a scaling out/in decision based on the number of messages to be processed. In the following screenshot you can see how uploading multiple photos in the S3 bucket has triggered the metric alarm to transition from Insufficient data to In alarm and finally to OK as the workload has been processed.
We will continue to implement a sustainable approach by deploying our AWS Graviton based queue workers using attribute-based instance type selection - the amount of memory and computing power that we need for the workload planned to run on the instances. Among others, this strategy has the advantage of simplifying the instance type selection process. Notice the CpuManufacturers
option. Moreover, we will utilize conditions like AllowedInstanceTypes
and ExcludedInstanceTypes
to select or avoid entire instance families. This is implemented in the CloudFormation template included in this repository.
This workshop demonstrates how to architect fault tolerant worker tiers in a sustainable way. If your queue worker tiers and workload needs are fault tolerant, you can increase your application’s efficiency and sustainability by implementing a queue-driven design. The main best practices covered in this lab are:
- Optimize systems to provision and consume only the resources needed for each workload.
- Scale up compute using auto-scaling services for processing messages asynchronously. This allows consuming resources to be right-sized.
- Use message queuing if you have relaxed availability requirements, reducing the infrastructure needed for workers running all the time.
To cleanup this lab, open the CloudFormation console and:
- Ensure you select the same region you used to deploy this workshop.
- Select the CloudFormation stack you used and click Delete.
See CONTRIBUTING for more information.
This library is licensed under the MIT-0 License. See the LICENSE file.