Scan Server ZooKeeper entries are not removed on shutdown.
Closed this issue · 3 comments
The scan servers write a lock with a path similar to: /accumulo/[IID]/sservers/host:port/zlock#[LOCK_UUID]#[LOCK_SEQ]
The lock nodes are ephemeral and are removed if the server cannot renew the lease. However, the host:port part of the path is persistent, even across ZooKeeper restarts. The .../sservers/host:port path should be removed if a scan server is shutdown and there should be a cleaner thread that removes scan servers that do not have a lock if the scan server was not shutdown cleanly.
There should be similar code that is used for teservers that maybe could be reused or adopted.
This may apply to other services like external compactors, but not sure.
#4560 is a duplicate, so closing it. It has some code links for the tsever and compactor code. Could not find anything for scan servers.
The .../sservers/host:port path should be removed if a scan server is shutdown
there should be a cleaner thread that removes scan servers that do not have a lock if the scan server was not shutdown cleanly.
Agreed, sounds like these are missing. As an aside, ZooZap can clean them up, and does so on a shutdown when using accumulo-cluster, ref here.
CompactionCoordinator.cleanUpCompactors performs clean up of ZooKeeper entries for Compactors that no longer exist. LiveTServerSet.startListeningForTabletServerChanges which is called from the Manager and the CompactionCoordinator remove ZooKeeper entries for TabletServers that no longer exist. Currently only the Client code and ZooKeeper have any reference to ScanServers. No other Accumulo process is aware of them. I can add something in the Manager that does the clean up.