Elastic Search Split Brain Issues (aka multiple masters)

[Reuenthal767]

This issue is related to all Elastic Search versions prior to 7 (so it affects all our fury logging modules to date).
From version 7 onwards this won't be a problem anymore since the cluster coordination module has ben reworked: https://www.elastic.co/blog/a-new-era-for-cluster-coordination-in-elasticsearch

ElasticSearch is a very flexible system that can be configured with (nearly) any number of nodes and master nodes. How does ElasticSearch know how many nodes are needed to reach a quorum for master election? By taking into account a setting called minimum_master_nodes.

We can get the current configured setting with the following api call:
GET _cluster/settings?include_defaults&filter_path=**.minimum_master_nodes

This is the result:
"discovery.zen.minimum_master_nodes" : "-1"

"-1" it's the same as 1, which means that in case of a network segmentation or in case of an unfortunate restart of the pod where the current master node lives the ES cluster can end up having 2 masters (split brain), a situation that makes the cluster unusable and that if it drags for too much can create dangling indices and other problems (this happened a lot of times to me). The only way that I know of to recover from a split brain is to scale down the ES sts to 0 replicas and then back again to 3.

To solve this problem we need to make 2 changes:

• set "discovery.zen.minimum_master_nodes" : "2" for elasticsearch triple
• rework the readiness probe accordingly (until there are 2 nodes up the health api endpoint does not return success, but if the readiness probe is not returning success the pod won't start accepting traffic and the next pod of the sts won't come up, setting sts podManagementPolicy: "Parallel" won't help either because if the pods cannot accept traffic they won't be able to form a cluster)

[angelbarrera92]

I am updating the module to provide ES7. So these issues are going to be resolved in the upcoming release.

[nandajavarma]

Closing since the issue is fixed with latest release.