rancher/prometheus-federator

Rancher-project scoped AlertmanagerConfig is NOT loaded as a config in alertmanager pod

Closed this issue · 4 comments

Labidi-github commented

Cluster Setup

  • Kubernetes version:
  • Type of Cluster : EKS
  • Installation :
    • rancher-monitoring version: 100.1.2+up19.0.3 ==> prometheus-operator version v0.50.0
    • prometheus-federator version v0.1.0

Describe the bug
As a project owner, i create a 'Project Monitors' inside my project 'my-project'. Every thing seems to work perfectly. Prometheus, Alertmanager and Grafana instances are deployed in 'cattle-my-project-id-monitoring' namespace (only Cluster Admins have access to this namespace).
When creating an AlertmanagerConfigs object (routes and receivers), config is not loaded in alertmanager pod !

As a project owner i can't do anything, since the stack is deployed in a namespace that I don't have access to !!!

As cluster Admin i'm able to force Alertmanager object config to load this config in a secret but i' don't think that's the good way !!

aiyengar2 commented

@Labidi-github thanks for filing this issue.

FYI, re:

As cluster Admin i'm able to force Alertmanager object config to load this config in a secret but i' don't think that's the good way !!

Based on the RBAC permissions assigned in:

prometheus-federator/charts/rancher-project-monitoring/0.1.0/templates/dashboard-roles.yaml

Line 89 in 132d866

- {{ printf "%s-alertmanager-secret" (include "project-prometheus-stack.fullname" .) }}

If you have edit permissions in the registration namespace (which should be provided to all Project Members), once the secret is created you do have the ability to modify it without cluster admin permissions. You just can't necessarily easily create it on the Rancher UI.

aiyengar2 commented

The config not being loaded into the Alertmanager pod is a weird bug. @ronhorton did we test this use case on release?

aiyengar2 commented

@Labidi-github, I looked into this with @ronhorton and was able to observe that the creation of an AlertmanagerConfig in a project namespace (e.g. default) did result in a modification of the Alertmanager configuration (which is found in the Status page of the Alertmanager UI) in a normal RKE cluster. We did not check whether this is an issue isolated to an EKS cluster, although I believe it is unlikely that that is the case.

Important: Could you provide more steps to reproduce your setup? Also, could you copy and paste the YAML for the Alertmanager CR created in your release namespace (cattle-project-<id>-monitoring) as well as the configuration found on the Status page of the Alertmanager UI?

Possible issues / gotchas:

  • If you tested whether the configuration "worked" by testing if an alert is sent, it should be noted that routes / receivers configured with an AlertmanagerConfig will only send alerts if the PromQL query used to construct the alert value has a namespace label whose value matches the namespace the AlertmanagerConfig is created within. So for example, an AlertmanagerConfig will not send the alert if the PrometheusRule defined to send the alert was vector(1) unless you added the namespace as an additional label that needs to be sent. ref: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#alertmanagerconfigspec
  • The AlertmanagerConfig will only be picked up by a Project Alertmanager if it resides within a Project Namespace since the Alertmanager CR per project is only defined to pick up namespaces that have the project label attached to them:

    prometheus-federator/charts/rancher-project-monitoring/0.1.0/templates/alertmanager/alertmanager.yaml

    Line 59 in cccd595

    alertmanagerConfigNamespaceSelector: {{ .Values.global.cattle.projectNamespaceSelector | toYaml | nindent 4 }}
    . This means that to set up the alerting, you will need to create a PrometheusRules in a project namespace (not cattle-monitoring-system, so that the alerting rule is picked up by your Project Prometheus, not your Cluster Prometheus) that has the label namespace in the outputted series as well as an AlertmanagerConfig in a project namespace (so that the alerting configuration is picked up by your Project Alertmanager, not your Cluster Alertmanager) to ensure that everything works as expected.
MKlimuszka commented

Since the reporter hasn't added more info. I am closing this as stale. It can be reopened if needed.