[metrics]: AWS/SNS: Metrics NumberOfNotificationsDelivered/NumberOfNotificationsFailed do not get exported

Question

[metrics]: AWS/SNS: Metrics NumberOfNotificationsDelivered/NumberOfNotificationsFailed do not get exported

danielrolfes2307 opened this issue a year ago · 8 comments

Context information

AWS service: SNS
CloudWatch namespace: AWS/SNS
Link to metrics documentation for this service: https://docs.aws.amazon.com/sns/latest/dg/sns-monitoring-using-cloudwatch.html
AWS region of the exporter: eu-central-1
AWS region of the service: eu-central-1

Exporter configuration

  ###############################################
  # SNS metrics
  ###############################################
  - aws_namespace:   AWS/SNS
    aws_metric_name: NumberOfNotificationsDelivered
    aws_dimensions:  [TopicName,PhoneNumber]
    aws_statistics: [Sum]
    set_timestamp: false
  - aws_namespace:   AWS/SNS
    aws_metric_name: NumberOfNotificationsFailed
    aws_dimensions:  [TopicName,PhoneNumber]
    aws_statistics: [Sum,Average]
    set_timestamp: false
  - aws_namespace:   AWS/SNS
    aws_metric_name: NumberOfMessagesPublished
    aws_dimensions:  [TopicName]
    aws_statistics: [Sum]
    set_timestamp: false
  - aws_namespace:   AWS/SNS
    aws_metric_name: PublishSize
    aws_dimensions:  [TopicName]
    set_timestamp: false

Exporter logs

What do you expect to happen?

"NumberOfNotificationsDelivered" and "NumberOfNotificationsFailed" should be exported.

What happened instead?

Although I get datapoints in Cloudwatch metrics for "NumberOfNotificationsDelivered" and "NumberOfNotificationsFailed" they do not show up when calling metrics-endpoint of cloudwatch-exporter via: curl localhost:9095/metrics

Metrics "NumberOfMessagesPublished" and "PublishSize" DO show up though.

Answer 1 · 2023-10-25T13:06:10.000Z

Update: I just ran cloudwatch-exporter with "warn_on_empty_list_dimensions: true"
"ignoring metric AWS/SNS:NumberOfNotificationsFailed due to dimensions mismatch"
I'm unfortunately not as familiar with Cloudwatch metrics/Cloudwatch exporter. Could someone help me out? :)

Answer 2 · 2023-11-16T08:56:57.000Z

Any help would be greatly appreciated :)

Answer 3 · 2024-04-16T15:35:08.000Z

Hi @matthiasr
I would like to try to solve this problem, can you assign this task to me,thanks.

Answer 4 · 2024-04-17T13:55:43.000Z

Hi @danielrolfes2307

I reproduced the issus,
below is my result

`# HELP aws_sns_number_of_messages_published_sum CloudWatch metric AWS/SNS NumberOfMessagesPublished Dimensions: [TopicName] Statistic: Sum Unit: Count

TYPE aws_sns_number_of_messages_published_sum gauge

aws_sns_number_of_messages_published_sum{job="aws_sns",instance="",topic_name="test_test",} 1.0
aws_sns_number_of_messages_published_sum{job="aws_sns",instance="",topic_name="test",} 2.0

HELP aws_sns_publish_size_sum CloudWatch metric AWS/SNS PublishSize Dimensions: [TopicName] Statistic: Sum Unit: Bytes

TYPE aws_sns_publish_size_sum gauge

aws_sns_publish_size_sum{job="aws_sns",instance="",topic_name="test_test",} 354.0
aws_sns_publish_size_sum{job="aws_sns",instance="",topic_name="test",} 6404.0

HELP aws_sns_publish_size_sample_count CloudWatch metric AWS/SNS PublishSize Dimensions: [TopicName] Statistic: SampleCount Unit: Bytes

TYPE aws_sns_publish_size_sample_count gauge

aws_sns_publish_size_sample_count{job="aws_sns",instance="",topic_name="test_test",} 1.0
aws_sns_publish_size_sample_count{job="aws_sns",instance="",topic_name="test",} 2.0

HELP aws_sns_publish_size_minimum CloudWatch metric AWS/SNS PublishSize Dimensions: [TopicName] Statistic: Minimum Unit: Bytes

TYPE aws_sns_publish_size_minimum gauge

aws_sns_publish_size_minimum{job="aws_sns",instance="",topic_name="test_test",} 354.0
aws_sns_publish_size_minimum{job="aws_sns",instance="",topic_name="test",} 3180.0

HELP aws_sns_publish_size_maximum CloudWatch metric AWS/SNS PublishSize Dimensions: [TopicName] Statistic: Maximum Unit: Bytes

TYPE aws_sns_publish_size_maximum gauge

aws_sns_publish_size_maximum{job="aws_sns",instance="",topic_name="test_test",} 354.0
aws_sns_publish_size_maximum{job="aws_sns",instance="",topic_name="test",} 3224.0

HELP aws_sns_publish_size_average CloudWatch metric AWS/SNS PublishSize Dimensions: [TopicName] Statistic: Average Unit: Bytes

TYPE aws_sns_publish_size_average gauge

aws_sns_publish_size_average{job="aws_sns",instance="",topic_name="test_test",} 354.0
aws_sns_publish_size_average{job="aws_sns",instance="",topic_name="test",} 3202.0
`
with the config:

region: us-east-1
metrics:

aws_namespace: AWS/SNS
aws_metric_name: NumberOfNotificationsDelivered
aws_dimensions: [TopicName,PhoneNumber]
aws_statistics: [Sum]
set_timestamp: false
aws_namespace: AWS/SNS
aws_metric_name: NumberOfNotificationsFailed
aws_dimensions: [TopicName,PhoneNumber]
aws_statistics: [Sum,Average]
set_timestamp: false
aws_namespace: AWS/SNS
aws_metric_name: NumberOfMessagesPublished
aws_dimensions: [TopicName]
aws_statistics: [Sum]
set_timestamp: false
aws_namespace: AWS/SNS
aws_metric_name: PublishSize
aws_dimensions: [TopicName]
set_timestamp: false

From the Config, I find that PhoneNumber dimensions, I think the metric NumberOfNotificationsFailed and NumberOfNotificationsDelivered doesn't have PhoneNumber dimensions. Please check your metrics.
below is my metric which doesn't PhoneNumber dimensions.

Answer 5 · 2024-04-19T11:08:57.000Z

@LiuQhahah
Thanks for looking into it...Appreciate it :)
Will try your configuration.
According to the documentation (https://docs.aws.amazon.com/sns/latest/dg/sns-monitoring-using-cloudwatch.html#sns-metric-dimensions) ~~the PhoneNumber will show up, if you configure SNS to publish these events as SMS.~~ (see EDIT)
Which is what we do in our case. Should have clarified that in my initial message :)

EDIT:
Just realized that this is only the case when we send the event to only ONE phone number, we send them to multiple, so the Dimension "PhoneNumber" might actually not be there.
I will check and get back to you asap :)

Answer 6 · 2024-04-24T08:04:25.000Z

@LiuQhahah
I had the chance to look into this now. We use the "Direct Push" scenario regarding SNS/SMS, where we are interested in getting the info for NumberOfNotificationsDelivered/NumberOfNotificationsFailed:

When producing a "failed SMS delivery" I do get Datapoints in Cloudwatch metrics:

Hovering over the graphed metrics in Cloudwatch metrics I see following details:
Namespace
AWS/SNS
Metric name
NumberOfNotificationsDelivered
PhoneNumber
PhoneNumberDirect

Namespace
AWS/SNS
Metric name
NumberOfNotificationsFailed
PhoneNumber
PhoneNumberDirect

Namespace
AWS/SNS
Metric name
NumberOfMessagesPublished
TopicName
SOME_ACCOUNT_NUMBER:phone

Based on this information I use following Cloudwatch exporter config:

aws_namespace: AWS/SNS
aws_metric_name: NumberOfNotificationsDelivered
aws_dimensions: [PhoneNumber]
aws_statistics: [Sum]
set_timestamp: false
aws_namespace: AWS/SNS
aws_metric_name: NumberOfNotificationsFailed
aws_dimensions: [PhoneNumber]
aws_statistics: [Sum]
set_timestamp: false
aws_namespace: AWS/SNS
aws_metric_name: NumberOfMessagesPublished
aws_dimensions: [TopicName]
aws_statistics: [Sum]
set_timestamp: false

Cloudwatch-Exporter is giving me following errors:
│ Apr 24, 2024 7:47:15 AM io.prometheus.cloudwatch.DefaultDimensionSource listDimensions │
│ WARNING: (listDimensions) ignoring metric AWS/SNS:NumberOfNotificationsDelivered due to dimensions mismatch │
│ Apr 24, 2024 7:47:16 AM io.prometheus.cloudwatch.DefaultDimensionSource listDimensions │
│ WARNING: (listDimensions) ignoring metric AWS/SNS:NumberOfNotificationsFailed due to dimensions mismatch │
│ Apr 24, 2024 7:47:16 AM io.prometheus.cloudwatch.DefaultDimensionSource listDimensions │
│ WARNING: (listDimensions) ignoring metric AWS/SNS:NumberOfMessagesPublished due to dimensions mismatch

Hope this helps :)

Answer 7 · 2024-04-24T14:54:30.000Z

Hi @danielrolfes2307

I try to reproduce the issue. When I add the PhoneNumber dimension, I found the result is empty from the response (see pic below). Can you debug it and check if the response is empty?
Also, I found some "dimensions mismatch" issues in the repository, which might help you.
#504
#473
#442

the related file:
https://github.com/prometheus/cloudwatch_exporter/blob/master/src/main/java/io/prometheus/cloudwatch/DefaultDimensionSource.java#L91

Answer 8 · 2024-04-24T15:45:34.000Z

@LiuQhahah
Thanks for linking the issues. I think I got carried away with the "WARNING: (listDimensions) ..." in Cloudwatch exporter logs.
I checked in Prometheus and all of the desired metrics have been properly exported with the aforementioned config:
aws_namespace: AWS/SNS
aws_metric_name: NumberOfNotificationsDelivered
aws_dimensions: [PhoneNumber]
aws_statistics: [Sum]
set_timestamp: false
aws_namespace: AWS/SNS
aws_metric_name: NumberOfNotificationsFailed
aws_dimensions: [PhoneNumber]
aws_statistics: [Sum]
set_timestamp: false
aws_namespace: AWS/SNS
aws_metric_name: NumberOfMessagesPublished
aws_dimensions: [TopicName]
aws_statistics: [Sum]
set_timestamp: false

Thanks a lot for looking into this! :)