Riak client does not support the use of a load balancer

Question

Riak client does not support the use of a load balancer

Closed this issue 8 years ago · 14 comments

We are using a load balancer (an F5) in our environment to proxy requests to the Riak servers. Our configuration looks like this:

<riakConfig nodePollTime="5000" defaultRetryWaitTime="200" defaultRetryCount="3">
  <nodes>
    <node name="dev1" hostAddress="RIAK-LB" pbcPort="8087" poolSize="10" />
  </nodes>
</riakConfig>

The load balancer sits in front of 5 Riak servers. Under normal operation this works correctly. However, when we stop one of the Riak nodes, clients that have a connection to that server mark the node 'dev1' as 'inactive'. Since this is the only node available, further calls result in an error of "Cluster Unavailable" being returned. This error continues until the background polling thread detects that a connection can be made, which can take up to 5 seconds.

We have also tried setting the configuration like this:

<riakConfig nodePollTime="5000" defaultRetryWaitTime="200" defaultRetryCount="3">
  <nodes>
    <node name="dev1" hostAddress="RIAK-LB" pbcPort="8087" poolSize="10" />
    <node name="dev2" hostAddress="RIAK-LB" pbcPort="8087" poolSize="10" />
    <node name="dev3" hostAddress="RIAK-LB" pbcPort="8087" poolSize="10" />
    <node name="dev4" hostAddress="RIAK-LB" pbcPort="8087" poolSize="10" />
    <node name="dev5" hostAddress="RIAK-LB" pbcPort="8087" poolSize="10" />
  </nodes>
</riakConfig>

In this case we'll only get the cluster unavailable if all 5 nodes are connected to the same physical machine. Since our load balancer uses a round robin this is unlikely, but it's not guaranteed to work because requests from other clients on the network also hit the load balancer.

Answer 1 · 2016-04-13T01:35:58.000Z

@rob-somerville you already tried the usual workaround. Thanks for the PR, I'll check it out.

Answer 2 · 2016-04-15T19:28:57.000Z

Hi @rob-somerville -

I pushed a new beta release: https://www.nuget.org/packages/RiakClient/2.3.0-beta2

Please note that the cluster configuration setting is externalLoadBalancer:

<riakConfig externalLoadBalancer=""true"">
...
...
...
</riakConfig>

Only configure one node, pointing to your load balancer, and the client won't mark it offline.

If it's convenient to test in your environment, please do so and fill me in on the results.

Thanks and have a great weekend -
Luke

Answer 3 · 2016-04-15T22:14:18.000Z

Thanks Luke,

We’ll pull it down and give it a try.

Thanks again.

Rob

From: Luke Bakken [mailto:notifications@github.com]
Sent: Friday, April 15, 2016 12:29 PM
To: basho/riak-dotnet-client riak-dotnet-client@noreply.github.com
Cc: Rob Somerville rob.somerville@mindbodyonline.com
Subject: Re: [basho/riak-dotnet-client] Riak client does not support the use of a load balancer (#306)

Hi @rob-somervillehttps://github.com/rob-somerville -

I pushed a new beta release:

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/306#issuecomment-210607485

Answer 4 · 2016-04-25T22:42:39.000Z

Hi @rob-somerville, any news from testing?

Answer 5 · 2016-04-26T16:41:32.000Z

Hey Luke,

So far it’s looking good. Tested with the load balancer and that looked good. More testing going on today and tomorrow as we integrate the beta into our latest code.

Thanks,

Rob

From: Luke Bakken [mailto:notifications@github.com]
Sent: Monday, April 25, 2016 3:43 PM
To: basho/riak-dotnet-client riak-dotnet-client@noreply.github.com
Cc: Rob Somerville rob.somerville@mindbodyonline.com; Mention mention@noreply.github.com
Subject: Re: [basho/riak-dotnet-client] Riak client does not support the use of a load balancer (#306)

Hi @rob-somervillehttps://github.com/rob-somerville, any news from testing?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/306#issuecomment-214553063

Answer 6 · 2016-05-02T15:07:23.000Z

Hi Luke,

We finished up the last of our regression tests with the Beta client. We didn’t find any issues – looks good.

Thanks,

Rob

From: Luke Bakken [mailto:notifications@github.com]
Sent: Monday, April 25, 2016 3:43 PM
To: basho/riak-dotnet-client riak-dotnet-client@noreply.github.com
Cc: Rob Somerville rob.somerville@mindbodyonline.com; Mention mention@noreply.github.com
Subject: Re: [basho/riak-dotnet-client] Riak client does not support the use of a load balancer (#306)

Hi @rob-somervillehttps://github.com/rob-somerville, any news from testing?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/306#issuecomment-214553063

Answer 7 · 2016-05-02T15:10:23.000Z

Thank you for the update. I'll get the final release out this week.

Answer 8 · 2016-05-17T18:25:01.000Z

Hey Luke,

Just checking in. Any update on the release?

Thanks,

Rob

From: Luke Bakken [mailto:notifications@github.com]
Sent: Monday, May 2, 2016 8:11 AM
To: basho/riak-dotnet-client riak-dotnet-client@noreply.github.com
Cc: Rob Somerville rob.somerville@mindbodyonline.com; Mention mention@noreply.github.com
Subject: Re: [basho/riak-dotnet-client] Riak client does not support the use of a load balancer (#306)

Thank you for the update. I'll get the final release out this week.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/306#issuecomment-216262758

Answer 9 · 2016-05-17T21:29:24.000Z

@rob-somerville - I have been swamped with other work, but will get to this today or tomorrow. Thanks for reminding me.

Answer 10 · 2016-05-18T01:09:00.000Z

@rob-somerville I am releasing 2.3.0-beta3 right now. It moves the externalLoadBalancer setting to the node configuration, from the cluster one. There are cases when people use multiple load balancers. My testing passes, if you could try it out I will get the final release out when I hear back. Thanks!!!

Answer 11 · 2016-05-19T18:39:13.000Z

Hey Luke,

We’ve updated our configs and verified. Looks good.

Thanks,

Rob

From: Luke Bakken [mailto:notifications@github.com]
Sent: Tuesday, May 17, 2016 6:09 PM
To: basho/riak-dotnet-client riak-dotnet-client@noreply.github.com
Cc: Rob Somerville rob.somerville@mindbodyonline.com; Mention mention@noreply.github.com
Subject: Re: [basho/riak-dotnet-client] Riak client does not support the use of a load balancer (#306)

@rob-somervillehttps://github.com/rob-somerville I am releasing 2.3.0-beta3 right now. It moves the externalLoadBalancer setting to the node configuration, from the cluster one. There are cases when people use multiple load balancers. My testing passes, if you could try it out I will get the final release out when I hear back. Thanks!!!

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/306#issuecomment-219898866

Answer 12 · 2016-05-19T19:03:30.000Z

Awesome! Official release on its way.

Answer 13 · 2016-05-19T20:00:52.000Z

@rob-somerville - you're all set. Thank you for your patience and willingness to test.

https://www.nuget.org/packages/RiakClient/

Answer 14 · 2016-05-19T20:10:27.000Z

Thanks Luke, appreciate it.

From: Luke Bakken [mailto:notifications@github.com]
Sent: Thursday, May 19, 2016 1:09 PM
To: basho/riak-dotnet-client riak-dotnet-client@noreply.github.com
Cc: Rob Somerville rob.somerville@mindbodyonline.com; Mention mention@noreply.github.com
Subject: Re: [basho/riak-dotnet-client] Riak client does not support the use of a load balancer (#306)

@rob-somervillehttps://github.com/rob-somerville - you're all set. Thank you for your patience and willingness to test.

https://www.nuget.org/packages/RiakClient/

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/306#issuecomment-220435780