spotify/folsom

Close connections after DNS update

Closed this issue · 4 comments

Hi,

Looking at both SrvKetamaClient and SrvKetamaClientTest and it looks likeupdateDNS only does additive changes. I mean, if the DNS returns a different list of hosts, the connection to the previous hosts will be maintained until the connections get severed by the hosts.

Is this right? If not, will it be possible to close the connections to the previous hosts when this occurs?

I think I misread SrvKetamaClientTest. You do check that connections are closed after the DNS update. Though, what's the expected behavior when the DNS resolver returns an empty list?

When the DNS is updated to a new set of hosts, it will try to connect to all the hosts in the new list.
Once it has successfully connected to the new hosts, it will schedule a job to clean up the old connections.

If the DNS list is empty that could be a problem, but it would be the same case as if you restarted the application and thus the SrvKetamaClient without any hosts.

I am not sure if it's a good idea to protect against empty lists in DNS - it would just hide the problem until later (unless DNS is glitchy and randomly returns empty lists for short durations, which would be odd).

FWIW, we've seen and guarded against glitchy DNS occasionally returning empty lists of hosts for some queries in the Spotify backend. See https://github.com/spotify/dns-java/blob/master/src/main/java/com/spotify/dns/RetainingDnsSrvResolver.java

There is now some protection against empty lists.