Logstash Exporter and multiple logstash pods
alekslebedev opened this issue · 4 comments
Hi! I have multiple Logstash pods within my Kubernetes cluster. I've set up a ClusterIP service for them and configured Logstash exporter on this service. However, I have some doubts. Will the metrics be duplicated, and will they be correct in this setup? Moreover, will the Grafana dashboard function correctly? Essentially, am I configuring the dashboard for Logstash exporter or for the Logstash instances themselves?
Hello @alekslebedev
since service is going to load balance the requests, you will essentially get metrics from a random pod each time.
What you can do is either to deploy exporter as a sidecar to each of the containers and tell prometheus/victoriametrics agent to scrape them (using labels if using k8s operator), or to deploy logstash pods with predictable dns names (statefulset does this out of the box) and configure logstash exporter to scrape all of them. please note, that the second option is only possible using unstable (master) version of the exporter, while sidecar can be created with either version, but requires custom helm configuration. currently i am thinking about creating a k8s operator to simplify the process of monitoring multiple logstash instances in kubernetes, but it is pretty complex task and is currently blocked by another issue (config hot reload). since the exporter is my side project, i can't provide you with the expected time of the operator being ready
i just noticed you mentioned grafana dashboard. to be honest, you have to test it yourself, as i personally don't use it and it was created by another user, based on v1 version of the exporter. judging by screenshots shared by other people, it is supposed to work properly with multiple logstash instances
Thanks for the quick answer! Now it is clear.