prometheus/cloudwatch_exporter

[bug]: Not getting all metrics for multiple resources with tags

shubhadaR opened this issue · 3 comments

shubhadaR commented

We are trying to fetch metrics with tags (using aws_tag_select) for multiple resources.
However we are getting metrics only for 2 or 3 resources but not for all. Also we are getting aws_resource_info only for few resources at a time but not for all resources.
here is our config file:

region: ap-south-1
#delay_seconds: 10
#period_seconds: 600
metrics:

  • aws_namespace: AWS/SQS
    use_get_metric_data: false
    aws_metric_name: ApproximateNumberOfMessagesNotVisible
    aws_dimensions: [QueueName]
    aws_tag_select:
    tag_selections:
    Monitoring: ["enabled"]
    resource_type_selection: "sqs:queue"
    resource_id_dimension: QueueName
    aws_statistics: [Average]

  • aws_namespace: AWS/ELB
    use_get_metric_data: false
    aws_metric_name: RequestCount
    aws_dimensions: [AvailabilityZone, LoadBalancerName]
    aws_tag_select:
    tag_selections:
    Monitoring: ["enabled"]
    resource_type_selection: "elasticloadbalancing:loadbalancer"
    resource_id_dimension: LoadBalancerName
    aws_statistics: [Sum]

  • aws_namespace: AWS/RDS
    use_get_metric_data: false
    aws_metric_name: CPUUtilization
    aws_dimensions: [DBInstanceIdentifier]
    aws_tag_select:
    tag_selections:
    Monitoring: ["enabled"]
    resource_type_selection: "rds:db"
    resource_id_dimension: DBInstanceIdentifier
    aws_statistics: [Sum]

  • aws_dimensions:

    • InstanceId
      aws_metric_name: CPUUtilization
      aws_namespace: AWS/EC2
      aws_statistics:
    • Average
      aws_tag_select:
      tag_selections:
      Monitoring: ["enabled"]
      resource_type_selection: "ec2:instance"
      resource_id_dimension: InstanceId

  • aws_namespace: AWS/ApplicationELB
    use_get_metric_data: false
    aws_metric_name: RequestCount
    aws_dimensions: [AvailabilityZone, LoadBalancer]
    aws_tag_select:
    tag_selections:
    Monitoring: ["enabled"]
    resource_type_selection: "elasticloadbalancing:loadbalancer"
    resource_id_dimension: LoadBalancer

matthiasr commented

That's concerning 😬 and difficult to debug without access to your environment. What happens when you look for these resources using aws resourcegroupstaggingapi get-resources?

The core of fetching the resources is here. Could you add more debug logging there to figure out where it's going wrong?

matthiasr commented

I believe this is another instance of #268. You may be interested in #571 and #648.