HA Entropy Manager

[allingeek]

To run Entropy in HA mode we're going to need to elect a managing control loop. That will require mutual awareness and either direct co-integration via something like Raft, or racing for a lock.

I'm thinking that we can use one of the many scarce resources in a Swarm / whatever cluster. Container name would be simple enough.

Consider a mechanism by which each candidate manager races to create (and hold open a stream connection with) a container (as a named lock). If the connection to that container is lost, the lock is lost and the container should be auto-removed by the scheduler. The other candidate managers would - in reaction to the destroy event for the named container - race to create their own and win the election.

This is crude, but buys leadership election without the need for yet another clustering tech or an additional data store.