How to automatic fetch all the LoadBalancer (With Target Group) tags

[JesusFrontelo]

When you have a bunch of Load Balancers, and each one has multiple target groups, its really hard to manually specify tags with group_left (tag_xxx, tag_yyy, tag_zzz) instead of doing it with group_right, which automatically joins all tags to the metric.

Here i give you an example of my configuration to scrape metrics and a query, for use in graph or alerts.

First of all, you have to define wich metrics you want to collect, and pay attention on the dimension of each metric.

  - aws_namespace: AWS/ApplicationELB
    aws_metric_name: ActiveConnectionCount
    aws_dimensions: [LoadBalancer]
    aws_tag_select:
      resource_type_selection: elasticloadbalancing:loadbalancer/app
      resource_id_dimension: LoadBalancer
    aws_statistics: [Sum]

  - aws_namespace: AWS/ApplicationELB
    aws_metric_name: ConsumedLCUs
    aws_dimensions: [LoadBalancer]
    aws_statistics: [Average, Maximum, Minimum]

  - aws_namespace: AWS/ApplicationELB
    aws_metric_name: ProcessedBytes
    aws_dimensions: [LoadBalancer]
    aws_statistics: [Sum]

  - aws_namespace: AWS/ApplicationELB
    aws_metric_name: NewConnectionCount
    aws_dimensions: [LoadBalancer]
    aws_statistics: [Sum]

  - aws_namespace: AWS/ApplicationELB
    aws_metric_name: RequestCount
    aws_dimensions: [LoadBalancer]
    aws_statistics: [Sum]

  - aws_namespace: AWS/ApplicationELB
    aws_metric_name: HealthyHostCount
    aws_dimensions: [TargetGroup, LoadBalancer]
    aws_statistics: [Average, Minimum, Maximum]

  - aws_namespace: AWS/ApplicationELB
    aws_metric_name: UnHealthyHostCount
    aws_dimensions: [TargetGroup, LoadBalancer]
    aws_tag_select:
      resource_type_selection: elasticloadbalancing:targetgroup
      resource_id_dimension: TargetGroup
    aws_statistics: [Average, Minimum, Maximum]

  - aws_namespace: AWS/ApplicationELB
    aws_metric_name: TargetResponseTime
    aws_dimensions: [TargetGroup, LoadBalancer]
    aws_statistics: [Average]

See how i set the aws_tag_select section when the metric has only LoadBalancer Dimension, or when it has two dimensions, like UnHealthyHostCount, wich has LoadBalancer and TargetGroup dimensions.
With this settings you will recieve the aws_resource_info for only LoadBalancer dimension and for TargetGroup separately, something like:

LoadBalancer Dimension:
aws_resource_info{arn="arn:aws:elasticloadbalancing:eu-west-1:************:loadbalancer/app/MY-LB/MY-LB-ID", exported_job="aws_applicationelb", instance="localhost:PORT", job="The name of the job", load_balancer="app/MY-LB/MY-LB-ID", tag_Application="APP-CODE", tag_Domain="DOM", tag_Environment="PRO", tag_Product="PRODUCT-CODE", tag_Terraform="True"}

With TargetGroup Dimension:
aws_resource_info{arn="arn:aws:elasticloadbalancing:eu-west-1:************:targetgroup/MY-LB-TARGETGROUP-0/TargetGroup-ID", exported_job="aws_applicationelb", instance="localhost:PORT", job="The name of the job", tag_Application="APP-CODE", tag_Domain="DOM", tag_Environment="PRO", tag_Product="PRODUCT-CODE", tag_Terraform="True", target_group="MY-LB-TARGETGROUP-0/TargetGroup-ID"}

And here is the second part of the magic trick :P

You should change the value of the target_group label from the metric, because tag value is like 'target_group="targetgroup/MY-LB-TARGETGROUP-0/TargetGroup-ID", removing the first part of its value 'targetgroup/'. To do this you must use the function label_replace.

The query:
label_replace(aws_applicationelb_un_healthy_host_count_average, "target_group", "$1", "target_group", "targetgroup/(.+)") * on (target_group) group_right (load_balancer) (aws_resource_info{exported_job="aws_applicationelb"})

I hope I have explained it as well as possible, and that it will be useful for everyone.

[JesusFrontelo]

P.D: You can use this for NLB too. Only need to change the namespace of the metric from AWS/ApplicationELB to AWS/NetworkELB, and change the resource_type_selection, from :loadbalancer/app to :loadbalancer/net.

[matthiasr]

Oh this is awesome, thank you for the writeup! There is an example configuration for ALB, but it is very basic right now – could you extend it with your tricks and add some comments as explanation?

Instead of label_replace, one could also use metric relabeling to do this on ingestion, something like

- source_labels: [__name__,target_group]
  regex: 'aws_applicationelb_[^;]+;target_group/(.+)'
  target_label: target_group
  replacement: '$1'

(haven't tested this)

[JesusFrontelo]

thanks @matthiasr , i´ve tested what you say and the right config for it is:

      - source_labels: [target_group]
        regex: '^targetgroup/(.+)$'
        replacement: '$1'
        target_label: target_group

it´s a great thing to modify label values at job instead of replacing labels on each metric, thanks for the advice.

[JesusFrontelo]

Hi again @matthiasr ,

Do you have a special procedure to create a pull request? i mean, i.e do you need me to create a new branch with specific name?

[matthiasr]

Fork, branch, and create a PR. You will have to git commit --signoff for legal reasons. Generally it's easier for you to manage if you create a branch, rather than submitting from your fork's master.