This project contains source code that demos the following:
- How to integrate AWS OpenTelemetry with Java SpringBoot based microservices.
- Using AWS OpenTelemetry using Amazon ECS service pattern.
- How to send AWS OpenTelemetry metrics from ECS service to CloudWatch Metrics, AWS X-Ray and Amazon Managed Service for Prometheus.
- Create nested stacks using AWS SAM Nested applications.
The AWS Distro for OpenTelemetry (ADOT) is a secure, production-ready, AWS-supported distribution of the OpenTelemetry project.
This project shows how to add java OpenTelemetry agent with popular spring boot based microservices.
Refer the Dockerfile in the project to see how to add the java OpenTelemetry agent without making code changes to the application.
Several patterns can be used for deploying ADOT for observability. The common ones include
-
The sidecar pattern : A common practice in the observability world is to use sidecars to provide container instrumentation. The ADOT website shows how to configure the AWS OTel Collector to scrape metrics using the sidecar pattern. A main advantage of the sidecar pattern is that it is simple to configure and troubleshoot. However, when customers have thousands of microservices and multiple environments,compute cost of running sidecar per task can start to add up and this pattern may become expensive.
-
Amazon ECS service pattern: The Amazon ECS service deployment pattern is similar to the DaemonSet pattern in Kubernetes. An Amazon ECS service allows you to run and maintain a specified number of instances of a OTel task in ECS. If any of the tasks fail, the Amazon ECS service scheduler launches another task instance to replace the failed task. If Autoscaling is added to the OpenTelemetry service, it dynamically scales as more application services are added, and they push metrics to the OpenTelemetry service. AWS Service discovery is used with the OpenTelemetry service to provide a private namespace to the service, which can be used by application services to discover and push their metrics to OpenTelemetry service. The main advantage of this pattern is cost saving. Compute costs are reduced because the number of instrumentation containers no longer has a 1:1 relationship with the application containers. This project is demoing the pattern.
For more information on sidecar vs ecs service pattern please refer to this AWS Blog.
The OpenTelemetry configuration allows you to manage sources and destinations of metrics collected by the OTel.
Refer to ssm parameter in otel-service.yaml that contains the OpenTelemetry configuration used by the service to export metrics data to CloudWatch and Prometheus & traces data to X-Ray.
Once the project is deployed, you will be able to see 2 new namespaces in the CloudWatch metrics console.
You can also view AWS X-Ray service map
You can query Prometheus metrics using the these techniques.
Finally, instead of using Cloudformation nested stack to create infrastructure we are using AWS SAM Nested applications. Cloudformation nested stack can be hard to change locally and redeploy because it requires you to upload the updated templates to s3 bucket everytime you want to make a change. Instead, AWS SAM cli can manage that process for you. Though primarily used to create serverless infrastructure, AWS SAM nested application (AWS::Serverless::Application) can allow you to create nested stacks with a minor change to the parent Cloudformation template. AWS SAM will convert them to Cloudformation nested stacks internally before deploying.
WARNING: This project creates VPC, ALB, ECS Cluster & Services, CloudMap, S3 buckets that can incur cost. Cleanup resources once done working.
The AWS SAM CLI is an extension of the AWS CLI that adds functionality for building and testing Lambda applications. To use the AWS SAM CLI, you need the following tools:
- AWS SAM CLI - Install the AWS SAM CLI.
- Docker - Install Docker community edition.
To build and deploy your application for the first time, run the following in your shell:
sam build
sam deploy --guided --capabilities CAPABILITY_NAMED_IAMTo delete the sample pipeline that you created, use the AWS CLI. Assuming you used your project name for the stack name, you can run the following:
sam deleteSee the AWS SAM developer guide for an introduction to SAM specification, the SAM CLI, and serverless application concepts.
See CONTRIBUTING for more information.
See the LICENSE file.