Netflix/dyno

Duplicated hosts in custom HostSupplier implementation

Opened this issue · 0 comments

Observed weird/confusing behavior when implementing a dyno client into our Spring application.
Dynoclient is being initiated as a singleton service (using spring @service annotation). This could all very well be my misunderstanding of how spring's service annotation works under the hood, or just bad code in general. Looking to get a bit more information and understanding of what might be going on. I am suspicious that getHosts() is being called twice during the initialization of the client, and the hosts variable inside customHostSupplier is having the two hosts added twice. This all happened upon building the application and initializing the client.

Basic information:

  • Two hosts on two different racks, same datacenter
  • Before clearing the hosts (see the // hack comment in code below), we were hitting unexpected NullPointerExceptions around here (hostFromTokenMapSupplier was null)
  • We essentially had 4 hosts (one dupe of each actual host), so clearing out the hosts in my customHostSupplier forced it to have the correct number of hosts

Hoping to maybe gain some more knowledge on things, for all I know I could be doing something wrong. Let me know if you need any more information.

Disclaimer: Configurations seem to be all working and set up properly, things work with the hack in place

@Service
public class Dynosaur {
    private DynoJedisClient client;
    private static final Gson gson = new Gson();
    private Integer ttl;
    private Boolean enabled = true;

    public Dynosaur(
        // Params are coming from spring application.properties
        @Value("${dyno.servers}") List<String> servers,
        @Value("${dyno.racks}") List<String> racks,
        @Value("${dyno.port}") Integer port,
        @Value("${dyno.ttl}") Integer ttl,
        @Value("${dyno.token}") Long token,
        @Value("${dyno.enabled}") Boolean enabled
    ) {
        this.enabled = enabled;
        this.ttl = ttl;

        HostSupplier customHostSupplier = new HostSupplier() {
            private final List<Host> hosts = new ArrayList<>();

            @Override
            public List<Host> getHosts() {
                // The current hack to work around this
                hosts.clear();
                Integer i = 0;
                for (String server : servers) {
                    hosts.add(new Host(server, port, racks.get(i), Host.Status.Up));
                    i++;
                }
                return hosts;
            }
        };

        TokenMapSupplier tokenMapSupplier = new TokenMapSupplier() {
            Map<Host, HostToken> tokenMap = new HashMap<>();

            @Override
            public List<HostToken> getTokens(Set<Host> activeHosts) {
                activeHosts.forEach(activeHost -> tokenMap.put(activeHost, new HostToken(token, activeHost)));
                return new ArrayList<>(tokenMap.values());
            }

            @Override
            public HostToken getTokenForHost(Host host, Set<Host> activeHosts) {
                Set<Map.Entry<Host, HostToken>> set = tokenMap.entrySet();
                for (Object aSet : set) {
                    Map.Entry mentry = (Map.Entry) aSet;
                    if (((Host) mentry.getKey()).getHostName().equals(host.getHostName())) {
                        return (HostToken) mentry.getValue();
                    }
                }
                return null;
            }
        };

        String localRack = System.getProperty("LOCAL_RACK");

        ConnectionPoolConfigurationImpl cp = new ConnectionPoolConfigurationImpl("appname")
                .setFailOnStartupIfNoHosts(true)
                .setRetryPolicyFactory(new RetryNTimes.RetryFactory(2, true))
                .setLocalRack(localRack)
                .withTokenSupplier(tokenMapSupplier);

        this.client = new DynoJedisClient.Builder()
                .withApplicationName("appname")
                .withHostSupplier(customHostSupplier)
                .withCPConfig(cp)
                .build();
    }
}

Application properties that are being injected look something like this

dyno.servers=host1.abc.com,host2.abc.com
dyno.racks=rack1,rack2
dyno.port=8102
dyno.token=0
dyno.ttl=24
dyno.enabled=true

Invoking the dynosaur client was done through autowiring in spring in a service/controller, and would be called like

    @Autowired
    Dynosaur dynosaur;

    dynosaur.get("key");