Explanation of Service Discovery
Closed this issue · 6 comments
Could you explain me what exactly SD exposes? Does it expose services behind the agents or maybe exposes agents or something else?
Thank you.
The proxy SD options expose the Prometheus SD functionality described here: https://prometheus.io/docs/prometheus/latest/http_sd/
The link did not help me comprehend it. I understand how it works, but I am having trouble understanding it within the context of Prometheus proxy. If you could explain it to me with examples, it would be great.
Let's imagine we have a Kubernetes cluster. Suppose we have three ports open:
- cadvisor (kubelet): 10250/metrics/cadvisor
- Node exporter on each worker node (VM): 9100/metrics
- kube-state-metrics: 8080/metrics
We also have the agent running with access to these ports.
On the other side, we have the proxy. What should I use as SD so that the proxy can see all the targets and obtain metrics? Should I implement my own HTTP SD, or can I set up the Kubernetes API in the proxy's configuration?
What should SD return in as a result?
Ohh, I know what will help - an example of usage SD in the context of prometheus proxy.
Perhaps, I got the idea. SD in the proxy allows us not to edit prometheus.yml config in a section static_configs when new agents connect to the proxy. But before it becomes working I have to define a job with http_sd_configs in prometheus.yml
The SD endpoint reports the meta info for the targets that have been defined for the proxy to scrape.
I have never needed to use the service discovery endpoint, but users asked me to add it so I did.