arabold/aws-to-slack

Charts for pNN.NN Metrics fail to generate

Closed this issue · 4 comments

nason commented

Many of my alarms are using p99 statistics, and those seem to be causing the chart to fail with an error:

Error rendering chart: { InvalidParameterValue: The parameter Statistics.member.1 must be a value in the set [ Minimum, Maximum, Average, SampleCount, Sum ]

Have you seen https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricWidgetImage.html? Its not in the aws-sdk that comes in Lambda's Node runtime, but I think could be worth considering

homeyjd commented

The GetMetricWidgetImage is pretty cool -- hadn't seen that one before! Unfortunately, Slack only allows image_url and GetMetricWidgetImage returns a PNG as base64 text. This would require adding an S3 bucket requirement for the Lambda function. Life-cycle transitions could be used to delete the files after X months, but then Slack history stops showing images (if that matters?). S3 is also eventually-consistent, so there's a (slim) chance Slack would receive the message, send it to the client, and the client would see a 404 from S3.

The Google Charts API is old and deprecated, but it keeps all the image data compressed into the URL string, meaning there's no additional storage requirement. Fingers crossed Google doesn't delete the service for a while?

I've added preliminary support for pXX statistics here:
805d6b2
This should fix the initial issue -- would you be willing to test it out for me?

homeyjd commented

@nason After that fix, are your charts w/ p99 stats working?

homeyjd commented

Hoping it's fixed? Please re-open if not.

alberts-s commented

It appears that the issue is not resolved on b9e7cbe. I'm seeing exactly the same error message.

After taking a quick peek inside the relevant code it appears that the pXX metrics are not passed to getMetricStatistics via ExtendedStatistics argument. See relevant code here. (AWS JS SDK docs).

An example alarm trigger message coming from pXX alarm is as follows:

...
"Trigger": {
...
"StatisticType": "ExtendedStatistic",
"ExtendedStatistic": "p95",
...
}

I think the easiest way to resolve this would be to provide the ExtendedStatistic argument to query if StatisticType == "ExtendedStatistic".