Feature request: Multiple "hosts" for external elasticsearch
CoaxVex opened this issue · 17 comments
The clusterlogforwarder currently only supports writing to a single Elasticsearch URL.
The fluentd elasticsearch output plugin supports configuring multiple hosts though, what is what you would typically do when using an elasticsearch cluster. (https://docs.fluentd.org/output/elasticsearch#hosts-optional)
Would it be possible to support a configuration in the CLO that allows for configuring multiple external elasticsearch hosts?
The alternative is to use an external load balancing system, but that then complicates matching of the SSL certificates used by the elasticsearch nodes.
Can defining an additional output support your usecase? We are no longer adding features to our fluentd offering but if that is a choice for vector it would be something we would entertain
Can defining an additional output support your usecase?
No, that would just result in duplicate messages in elasticsearch. Elasticsearch clients are meant to configure a list of hosts and the client library then does the load balancing. The same goes for kafka, by the way.
No, that would just result in duplicate messages in elasticsearch. Elasticsearch clients are meant to configure a list of hosts and the client library then does the load balancing.
You could achieve load balancing additionally by defining a service that maps to multiple ES pods as well AFAIK that is one of the intents behind the design of the "service" resource.
Given vector, which is the collector we intend to use going forward does not appear to support multiple endpoints, I do not see us accepting your request until it does
The same goes for kafka, by the way.
Which it does https://github.com/openshift/cluster-logging-operator/blob/master/apis/logging/v1/output_types.go#L174
You could achieve load balancing additionally by defining a service that maps to multiple ES pods as well AFAIK that is one of the intents behind the design of the "service" resource.
Using a service could work if elasticsearch is running in pods on the same cluster, but when using clusterlogforwarding to an external cluster it will cause the SSL verification to fail unless the certificates used by elasticsearch are configured to include the name of the service (my-elasticsearch.my-namespace.svc.cluster.local) in a SAN field.
Given vector, which is the collector we intend to use going forward does not appear to support multiple endpoints, I do not see us accepting your request until it does
Actually, this got merged the same day you posted the update: vectordotdev/vector#14088
So I guess there is hope?
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle stale
/remove-lifecycle stale