[kube-prometheus-stack] global.ImageRegistry does not affect prometheus image for all instances

[omeryahud]

Describe the bug a clear and concise description of what the bug is.

When using the following values.yaml file, some images (specifically prometheus in my case) are pulled from upstream registries instead of the internal registry:

global:
  imageRegistry: registry.local/monitoring

The default prometheus instance's pod (within the stacks' namespace) pulls the image from the correct registry:

# kubectl get pod -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -oyaml | grep image:
    image: registry.local/monitoring/prometheus/prometheus:v2.52.0 # <----------------
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0
    image: registry.local/monitoring/prometheus/prometheus:v2.52.0 # <----------------
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0

However, instances in different namespaces still attempt to pull the upstream image:

# kubectl get pod -n test prometheus-test-0 -oyaml | grep image:
    image: quay.io/prometheus/prometheus:v2.52.0 # <----------------
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0
  - image: quay.io/prometheus/prometheus:v2.52.0 # <----------------
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0

After looking into the chart, and this template specifically: https://github.com/prometheus-community/helm-charts/blob/kube-prometheus-stack-59.1.0/charts/kube-prometheus-stack/templates/prometheus-operator/deployment.yaml#L88
I managed to workaround this issue by modifying my values.yaml file:

global:
  imageRegistry: registry.local/monitoring
prometheusOperator:
  prometheusDefaultBaseImage: prometheus/prometheus

which I don't think is the desired behavior

What's your helm version?

version.BuildInfo{Version:"v3.12.3", GitCommit:"3a31588ad33fe3b89af5a2a54ee1d25bfe6eaa5e", GitTreeState:"clean", GoVersion:"go1.20.7"}

What's your kubectl version?

Client Version: v1.29.6 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.6

Which chart?

kube-prometheus-stack

What's the chart version?

59.1.0

What happened?

Prometheus instances that reside in a different namespace than the stack pull upstream prometheus images even when configured to pull them from an internal registry

What you expected to happen?

All prometheus instances in the cluster to pull all of their images from the internal registry configured in the custom values.yaml file

How to reproduce it?

1. Run the following helm command:

helm upgrade -i kube-prometheus-stack --create-namespace -n monitoring ./kube-prometheus-stack --set global.imageRegistry="registry.local/monitoring"

2. Create a second Prometheus instance in a different namespace than the stack's namespace
3. Inspect the second instance's pods to see that the prometheus image is being pulled from upstream

Enter the changed values of values.yaml?

global:
  imageRegistry: registry.local/monitoring

Enter the command that you execute and failing/misfunctioning.

helm upgrade -i kube-prometheus-stack --create-namespace -n monitoring ./kube-prometheus-stack -f prom-values.yaml

Anything else we need to know?

No response

[jkroepke]

Are you interest in a PR?

[omeryahud]

Are you interest in a PR?

Sure I can take it, I'll get a PR ready in a couple of days