awslabs/aws-embedded-metrics-java

Support for AWS SDK V2 `MetricsPublisher`

cgledezma1101 opened this issue · 4 comments

cgledezma1101 commented

I have an application that uses AWS SDK V2 to perform multiple API calls, and that monitors those API calls using the recently released metrics module for AWS SDK V2

My application runs on Lambda, and uses EMF to avoid the pitfalls of calling CloudWatch APIs directly to emit monitoring metrics. Using this library we've been able to replace all of our CloudWatch API calls, except for the metrics emitted by the AWS SDK V2.

Because AWS' recommendation for metric tracking on ephemeral containers is to use EMF, it would be good to have integration in this library with the MetricsPublisher interface that the AWS SDK V2 exposes, so that we can monitor our AWS SDK calls with EMF as well.

I'm currently interested in Lambda support, but support for containers would also be good.

jaredcnance commented

I took a quick look at this to see what pre-requisites might be needed. It looks like the existing CloudWatch publisher does internal aggregation and publishes data either as a statistic set (summary aggregator) or values/counts arrays (detailed aggregator) on an interval. I think 2 things are required:

  1. Support for StatisticSets/Values+Counts in EMF (already planned). Not an absolute requirement, but simplifies the implementation.
  2. Async aggregation with graceful shutdown to ensure metrics get flushed when the Lambda function exits. This needs some more research, but one thought is that we simply expose a flush() method on the publisher and it would be the responsibility of the application developer to call flush before the function exits.
EMFMetricPublisher metricPublisher = EMFMetricPublisher.create();
DynamoDbClient dynamoDb = DynamoDbClient.builder()
                                        .overrideConfiguration(c -> c.addMetricPublisher(metricPublisher))
                                        .build();

// function code...

logger.flush();
metricPublisher.flush();
humanzz commented

See the related discussion in aws/aws-sdk-java-v2#2068 (comment)

The main comment/concern there was that every API call would lead to a System.out.println which might be a costly thing.

I wonder, if there's something can be done on the EMF library side here to support emitting multiple metrics loggers efficiently e.g. to minimize the calls to System.out.println.

This can be used in an EMFMetricPublisher implementation which exposes a flush method that then flushes the metrics efficiently. Calling the flush can be left as a responsibility of the application developer for now. (I also see that solutions like https://github.com/awslabs/aws-lambda-powertools-java can potentially build on top of this with its aspectj weaving to manage that)

jaredcnance commented

In most cases, writing to stdout should not be terribly expensive. It's only an issue (AFIAK) in environments with heavy multi-threading and very frequent writes. To reduce the cost of this, there are a few options:

  1. Use time aggregated values: https://github.com/aws/aws-sdk-java-v2/blob/master/metric-publishers/cloudwatch-metric-publisher/src/main/java/software/amazon/awssdk/metrics/publishers/cloudwatch/internal/transform/TimeBucketedMetrics.java#L45
  2. Add a queue between the logger and the sink and make the sink single-threaded. This eliminates contention on the stdout stream from the EMF library's perspective. This is needed anyways for the socket based clients but may be unnecessary overhead for most console based use cases.
cgledezma1101 commented

Any updates on this? This is still a requirement for my team to reduce our dependency in Lambda on CloudWatch.putMetricData