StateLeft members are never reaped
bwaters opened this issue · 3 comments
bwaters commented
Unlike StateDead members old nodes that gracefully leave are never reaped. They just permanently increase the node list and sit inside the push/pull state.
Eventually on a large cluster with lots of churn you will hit maxPushStateBytes and push pull stops working completely
bwaters commented
Looks like this bug was introduced when StateLeft was implemented.
237d410
Looks like most places in the code where == StateDead were updated to n.DeadOrLeft() but this one was missed.
From observation this is not occurring on consul 1.07 but is on 1.10.0 and 1.9.6
bwaters commented
Once a sufficient number of nodes reach the maxPushStateBytes limit then new members are unable to join the consul cluster