Question: What is/was driving idea to run multiple instances of Exhibitor ?
ksambhav opened this issue · 2 comments
After playing around a bit and going through the wiki I understand that on each machine that is part of Zookeeper ensemble, we need to have an instance Exhibitor running on that machine. : -
https://github.com/soabase/exhibitor/wiki/Installing
What was the driving force of this approach ?
What if we have just one instance of exhibitor ( possibly on a machine outside of zookeeper ensemble) that does all the supervising job. That single exhibitor instance can talk and manage( start, restart, stop, config files) remote Zookeeper instances over SSH.
When it comes to supervising ZooKeepers, the ZooKeeper Administrator's Guide states:
You will want to have a supervisory process that manages each of your ZooKeeper server processes (JVM). The ZK server is designed to be "fail fast" meaning that it will shutdown (process exit) if an error occurs that it cannot recover from. As a ZooKeeper serving cluster is highly reliable, this means that while the server may go down the cluster as a whole is still active and serving requests. Additionally, as the cluster is "self healing" the failed server once restarted will automatically rejoin the ensemble w/o any manual interaction.
Having a supervisory process such as daemontools or SMF (other options for supervisory process are also available, it's up to you which one you would like to use, these are just two examples) managing your ZooKeeper server ensures that if the process does exit abnormally it will automatically be restarted and will quickly rejoin the cluster.
So one of the jobs of the supervisor process is to quickly restart a ZooKeeper if it dies. A single instance of Exhibitor ssh'ing into 3 or 5 or however many ZooKeepers is simply untenable and also introduces a single point of failure, which is precisely what you're trying to avoid through the use of multiple ZooKeepers.
investigation complete