[kube-prometheus-stack] Prometheus defaultRules not limited to prometheusOperator / kube-state-metrics namespace list

[johnswarbrick-napier]

Describe the bug a clear and concise description of what the bug is.

I am deploying kube-prometheus-stack into a Kubernetes cluster with a large number of namespaces.

However I only want to monitor and alert on a small number of explicitly defined namespaces.

I have limited the namespace scope as follows:

kube-prometheus-stack:
  kube-state-metrics:
    releaseNamespace: true
    namespaces:
    - namespace1
    - namespace2
    - argo
    - ingress-nginx
    
  prometheusOperator:
    namespaces:
      releaseNamespace: true
      additional:
      - namespace1
      - namespace2
      - argo
      - ingress-nginx

But the Prometheus defaultRules are still matching other namespaces.

For example, I receive alerts for CPUThrottlingHigh from other namespaces not in the explicit list above.

The rule is:

sum by (cluster, container, pod, namespace) (increase(container_cpu_cfs_throttled_periods_total{container!=""}[5m])) / sum by (cluster, container, pod, namespace) (increase(container_cpu_cfs_periods_total[5m])) > (25 / 100)

How can I limit kube-prometheus-stack so the bundled Prometheus defaultRules are only triggered for the namespace list above?

I don't need to gather metrics or generate alerts from any other namespace within the cluster at all.

UPDATE:

I have discovered that the defaultRules are firing for metrics gathered by kubeletService

I think I need to somehow restrict kubeletService to only retrieve metrics for an explicit namespace list I provide.

Or I guess the other option is to filter out the kubeletService metrics for non-matching namespaces after they have been scraped. But this feels less efficient.

I'm just not sure how best to restrict/filter the namespaces that kubeletService is retrieving metrics for?

What's your helm version?

version.BuildInfo{Version:"v3.12.1", GitCommit:"f32a527a060157990e2aa86bf45010dfb3cc8b8d", GitTreeState:"clean", GoVersion:"go1.20.4"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647"

Which chart?

kube-prometheus-stack

What's the chart version?

58.2.2