Dynamic ZooKeeper Ensemble Support
Closed this issue · 10 comments
EDIT: This discussion ultimately led to discovering that ZooKeeper supports dynamic ensemble additions and subtractions with the right configuration and that a small change is required in RDF Delta to track those changes.
ZooKeeper's requirement to know about the other nodes at startup creates a surprising amount of difficulty when it comes to designing reproducible infrastructure for a cloud service because the identity and location of nodes are not known until after all the nodes have been bootstrapped. This requires the addition of more components that need to be fault-tolerant whether it be a DNS service that registers predetermined hostnames after the IP addresses are known or a process to dynamically update and restart the nodes based on configuration changes. Both are non-trivial solutions.
Consul provides a distributed Key/Value store similar to ZooKeeper, however, it uses a gossip protocol to discover other servers to form a cluster without any additional components. With some refactoring, it may be possible to provide users the option of one or the other. As it is, it may take me less time to produce a pull request for that than it will take me to design one of the aforementioned solutions to make ZooKeeper usable for my needs.
There are two ways of using ZooKeeper with RDF Delta. One way is to run embedded ZooKeeper, the other is a separate ZooKeeper layer.
What looks interesting for version 2 is to switch to use RAFT, specifically Apache Ratis. This is a library and so will be like using embedded Zookeeper at the moment, removing the additional admin of a separate ZooKeeper cluster. It should also be possible to clean up the abstract making alternative plugins easier.
Consul can not be co-resident with the patch server. Surely you still have to identify the servers in the SWIM pool for the Consul/Serf gossip protocol to operate?
Consul actually uses RAFT for consensus, so using it directly would eliminate the need for anything else. I approve of this direction.
It's not as important to me that ZooKeeper/Consul runs in the same process as RDF Delta as it is that ZooKeeper/Consul does not require an additional piece to get it to work in the first place. Consul does need to be pointed at the cluster, but it doesn't require complete knowledge of the entire cluster in each member. Basically, I can stand the first one up and point the second and third at the first without needing to provide connection info about every node. With ZooKeeper, I need them all to exist before I can generate a correct configuration file, but it's impossible to update the configuration after the instances have already been stood up without introducing an additional service of some kind or doing it by hand, thereby defeating the automation effort.
Consul uses RAFT - but with direct use of RAFT, the patch coordination could be done as a RAFT state machine.
I'm seeing now that ZooKeeper has a dynamic registration mode since version 3.5. I'm struggling to find where the embedded ZooKeeper is run in here, but I think we can just tweak it to register itself with a cluster to ease my problem in the near term. If you can help me find that bit of code, I'll see what I can do.
ZooServer zs = ZkS.runZookeeperServer(...)
I was able to get the embedded ZooKeeper server to dynamically join an ensemble with only the zoo.cfg and some ZooKeeper CLI commands. It will require a small change in the way the connection is established with Curator which I have documented here. I should have a pull request submitted for this change later today for your review. Thank you for your help, @afs.
Ensemble tracking defaults to enabled for current ZooKeeper.
https://issues.apache.org/jira/browse/ZOOKEEPER-3814 which is fixed with apache/zookeeper#1356 in ZooKeeper 3.7.0 is unrelated to this issue. The problem is not getting the members of the ZooKeeper Ensemble to dynamically update their config. That always worked. The problem is getting RDF Delta, a CLIENT of the ZooKeeper Ensemble, to dynamically update its config from /zookeeper/config. If you'd only actually test RDF Delta against a dynamically changing ZooKeeper Ensemble, it would be immediately obvious why I opened this ticket originally.
CuratorFrameworkFactory.builder.ensembleTracker.
You closed this citing an unrelated update to ZooKeeper, but now you're saying Curator does it. You haven't updated either of those dependencies, and I've reported to you that this, in fact, does not work. Reading the javadoc and making comments in the source code is no substitute for actually testing it, but you seem to be too stubborn to just do that. Oh, well. You can lead a horse to water, but you can't make him drink.