GoogleCloudPlatform/prometheus-engine

Can't create PodMonitoring on Autopilot 1.25

Closed this issue · 2 comments

Version is 1.25.15-gke.1115000. Get the following error:
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "default.podmonitorings.gmp-operator.gmp-system.monitoring.googleapis.com": failed to call webhook: Post "https://gmp-operator.gmp-system.svc:443/default/monitoring.googleapis.com/v1/podmonitorings?timeout=10s": service "gmp-operator" not found
I see the namespace is gke-gmp-system instead of gmp-system. I don't see a way to tell PodMonitoring what the operator namespace is. What the recommendation in this case?

I think it's working now. Turns out there were 2 webhooks gmp-operator.gke-gmp-system.monitoring.googleapis.com and gmp-operator.gmp-system.monitoring.googleapis.com (maybe leftover from an upgrade?). Deleting the gmp-system ones allowed me to create the PodMonitoring I was having trouble with :)

Hey Rafael, glad your issue got resolved.