[metrics]: Unable to set the provided role while deploying in Kubernetes environment

[abhisar007]

Context information

We are trying to deploy the cloudwatch exporter in EKS environment. When the deployment is successful, we see that scrape fails. As, it assumes the default cluster role and not the role with all permission provided in the values.yaml file during deployment.

I dont want to add the cloudwatch permission to the default cluster, as it is not recommended for production use. Please help if you came across similar issue. Any help is highly appreciated. I have added parts of my config.

• AWS service:
• CloudWatch namespace:
• Link to metrics documentation for this service:
• AWS region of the exporter:
• AWS region of the service:

Exporter configuration

aws:
  role: "arn:aws:iam::xx:role/xxx"
  # Enables usage of regional STS endpoints rather than global which is default
  stsRegional:
    enabled: false
  
secret:
    name: abcd
    
serviceAccount:
  # Specifies whether a ServiceAccount should be created
  create: true
  #false
  # The name of the ServiceAccount to use.
  # If not set and create is true, a name is generated using the fullname template
  name: "service_account_name"
  #default
  # annotations:
  # Will add the provided map to the annotations for the created serviceAccount
  # e.g.
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxx:role/xxxxx
    #eks.amazonaws.com/sts-regional-endpoints: "true"  
  # Specifies whether to automount API credentials for the ServiceAccount.
  automountServiceAccountToken: true

rbac:
  # Specifies whether RBAC resources should be created
  create: true

Exporter logs

software.amazon.awssdk.services.cloudwatch.model.CloudWatchException: User: arn:aws:sts::xxxxxxx:assumed-role/EKS/xxxxx is not authorized to perform: cloudwatch:ListMetrics because no identity-based policy allows the cloudwatch:ListMetrics action (Service: CloudWatch, Status Code: 403, Request ID: xxxxxxx)

What do you expect to happen?

What happened instead?

[Kavuti]

Hello, same problem here.
Although the aws.role property is meant to be used with kube2iam, the annotation should do work according to the Helm chart documentation here. Even though I set the annotation, I constantly see that the exporter is using the node role.
Actually, I would like to avoid to use the API Key, so it would be great to understand why this is happening.

[Kavuti]

Hi, I found a solution that is working for me.
I specified in the configuration of the cloudwatch-exporter the same role i used on the aws.role property.
I added the following root configuration property:
role_arn: arn:aws:iam::111111111111:role/my-cloudwatch-exporter-role
Since when I put the role, the exporter started retrieving the metrics.
If you are using the Helm chart you have to put it in the config value.
Hope it helps.

[matthiasr]

Thank you for helping out @Kavuti!