[metrics]: AmazonMQ some metrics missing

Question

[metrics]: AmazonMQ some metrics missing

stakot opened this issue 8 months ago · 1 comments

Context information

AWS service:
AWS/ActiveMQ
CloudWatch namespace:
AWS/ActiveMQ
Link to metrics documentation for this service:
https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/security-logging-monitoring-cloudwatch.html
AWS region of the exporter:
eu-central-1
AWS region of the service:
eu-central-1

Exporter configuration

  region: eu-central-1
  metrics:
  - aws_dimensions:
    - Broker
    aws_metric_name: TotalProducerCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Maximum

  - aws_dimensions:
    - Broker
    aws_metric_name: TotalConsumerCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Maximum

  - aws_dimensions:
    - Broker
    aws_metric_name: JournalFilesForFullRecovery
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Maximum

  - aws_dimensions:
    - Broker
    aws_metric_name: JournalFilesForFastRecovery
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Maximum

  - aws_dimensions:
    - Broker
    aws_metric_name: HeapUsage
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Average

  - aws_dimensions:
    - Broker
    aws_metric_name: CpuUtilization
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Average

  - aws_dimensions:
    - Broker
    aws_metric_name: TotalConsumerCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - SampleCount

  - aws_dimensions:
    - Broker
    aws_metric_name: StorePercentUsage
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Maximum

  - aws_dimensions:
    - Broker
    aws_metric_name: NetworkIn
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Average

  - aws_dimensions:
    - Broker
    aws_metric_name: NetworkOut
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Maximum

  - aws_dimensions:
    - Broker
    - Topic
    - Queue
    aws_metric_name: ConsumerCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Maximum

  - aws_dimensions:
    - Broker
    - Topic
    - Queue
    aws_metric_name: DispatchCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Maximum

  - aws_dimensions:
    - Broker
    - Topic
    - Queue
    aws_metric_name: ProducerCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Maximum

  - aws_dimensions:
    - Broker
    - Topic
    aws_metric_name: ReceiveCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Maximum

  - aws_dimensions:
    - Broker
    - Topic
    - Queue
    aws_metric_name: EnqueueCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Maximum

  - aws_dimensions:
    - Broker
    - Topic
    - Queue
    aws_metric_name: DequeueCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Maximum

  - aws_dimensions:
    - Broker
    - Topic
    - Queue
    aws_metric_name: ExpiredCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Maximum

  - aws_dimensions:
    - Broker
    - Topic
    - Queue
    aws_metric_name: EnqueueTime
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Maximum

  - aws_dimensions:
    - Broker
    - Topic
    aws_metric_name: MemoryUsage
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Average

  - aws_dimensions:
    - Broker
    aws_metric_name: TotalMessageCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Average

  - aws_dimensions:
    - Broker
    aws_metric_name: TotalEnqueueCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Average

  - aws_dimensions:
    - Broker
    aws_metric_name: TotalDequeueCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Average

  - aws_dimensions:
    - Queue
    aws_metric_name: QueueSize
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Average

  - aws_dimensions:
    - Broker
    - Queue
    aws_metric_name: DequeueCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Average

  - aws_dimensions:
    - Broker
    aws_metric_name: TotalConsumerCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Average

  - aws_dimensions:
    - Broker
    - Topic
    - Queue
    aws_metric_name: ConsumerCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Average

  - aws_dimensions:
    - Broker
    - Topic
    - Queue
    aws_metric_name: ProducerCount
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Average

  - aws_dimensions:
    - Broker
    aws_metric_name: BurstBalance
    aws_namespace: AWS/AmazonMQ
    aws_statistics:
    - Average

Exporter logs

Jan 18, 2024 4:17:09 PM org.eclipse.jetty.server.Server doStart
INFO: jetty-11.0.18; built: 2023-10-27T02:14:36.036Z; git: 5a9a771a9fbcb9d36993630850f612581b78c13f; jvm 17.0.9+9
Jan 18, 2024 4:17:09 PM org.eclipse.jetty.server.handler.ContextHandler doStart
INFO: Started o.e.j.s.ServletContextHandler@1bdaa23d{/,null,AVAILABLE}
Jan 18, 2024 4:17:09 PM org.eclipse.jetty.server.AbstractConnector doStart
INFO: Started ServerConnector@6fff253c{HTTP/1.1, (http/1.1)}{0.0.0.0:9106}
Jan 18, 2024 4:17:09 PM org.eclipse.jetty.server.Server doStart
INFO: Started Server@34abdee4{STARTING}[11.0.18,sto=0] @1170ms

What do you expect to happen?

All the metrics mentioned in config above appear in podhostname/metrics URI

What happened instead?

With config mentioned above I got only following metrics in /metrics tab (instead of JVM metrics):

"NetworkIn"
"TotalDequeueCount"
"MemoryUsage"
"TotalProducerCount"
"StorePercentUsage"
"TotalMessageCount"
"JournalFilesForFastRecovery"
"TotalEnqueueCount"
"NetworkOut"
"TotalConsumerCount"
"JournalFilesForFullRecovery"
"CpuUtilization"
"DequeueCount"
"HeapUsage"

I was trying to leave only one metric in config - QueueSize, and it wasn't returned anything except JVM metrics

P.S. Maybe you can point me where I can find some deeper log or how to enable some debug output so I can try to investigate it by myself.

P.P.S Exporter is running as pod in EKS cluster

Answer 1 · 2024-01-23T12:31:09.000Z

Ok, I found root cause of the my issue. It was wrong dimensions for metrics. For example QueueSize. In example it is set to "Queue" dimension, and, according to AWS documentation, it is correct. But in fact this metric is using two layers - Queue and Broker (this is confirmed by AWS CLI where you need to set both dimensions).
So my proposition to developers would be to set warn_on_empty_list_dimensions by default and add this string to all examples, so, at least, new users will know where to dig with empty metrics.