Is the sensu-prometheus-collector check scraping the right kube-state-metrics exporter?
Opened this issue · 2 comments
According to https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#communicating-with-daemon-pods (regarding DNS for DaemonSet Pods), the documentation states that you can "Create a service with the same Pod selector, and use the service to reach a daemon on a random node. (No way to reach specific node.)"
The kube-state-metrics project is configured to use a Service, so the check-kube-state-metrics.json check definition in this project would be hitting a random kube-state-metrics endpoint. Is this OK? Does kube-state-metrics expose cluster-level metrics, or does it provide metrics on a per node basis?
I believe the answer is that kube-state-metrics does generate cluster-level metrics, so hitting a random DaemonSet pod is acceptable. From the readme:
kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. (See examples in the Metrics section below.) It is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods.
From the testing ran last night, it does appear that when the error code (9) is occurring, that no metrics are being fed. Could this be a timing issue?