How to update tokens after adding nodes to cluster

Question

How to update tokens after adding nodes to cluster

Opened this issue 8 years ago · 11 comments

Hi,

Trying to find an answer in the archives, but only got an issue with "doubling the cluster" in wiki is empty.
Currently, we're evaluating Priam for our C* clusters in Xively and I'm wondering what's the easiest way of adding new nodes to a running cluster and especially how to do this properly in a multi-hundred-node cluster.

Working with manual token assignment (not using vnodes, because Priam doesn't support it), after adding a new node, I've to calculate new tokens and move nodes one-by-one. The quickest way looks like is that calling Priam's /move endpoint on each node and do the same with /cleanup as well.

Is there a better way implemented in Priam doing the same?

Question B is still: how to double the cluster size with Priam, because in that case token assignment is a bit easier?

Thanks,
Andor

Answer 1 · 2016-06-29T16:31:31.000Z

Hi Andor, will engage my colleagues at Netflix and get back to you.

Regards

Answer 2 · 2016-07-18T17:17:36.000Z

Hi Vinh,
Do you have an answer for this?

Answer 3 · 2016-07-19T16:31:35.000Z

I believe InstanceIdentity should be updated with the new token in SimpleDB by the /move endpoint, but according to the latest sources this is not the case.

Answer 4 · 2016-08-30T11:26:43.000Z

Hi Vinh,
Do you have an update on this thread?
Thanks,
Andor

Answer 5 · 2016-09-01T15:38:01.000Z

Hi Andy, sorry for the delay. I will try to spend time to research and
answer your questions.
-Vinh

On Tue, Aug 30, 2016 at 4:26 AM Andor Molnár notifications@github.com
wrote:

Hi Vinh,
Do you have an update on this thread?
Thanks,
Andor

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#493 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADNEbZXZnJcXRCDk2NaopmdXQ-3cTy7Tks5qlBN6gaJpZM4I4A8F
.

Answer 6 · 2016-11-11T01:55:29.000Z

Any luck on this? Currently we have to update SimpleDB manually.

Answer 7 · 2016-11-11T09:40:35.000Z

Hi BlueDragonX,

Yes that's still the case. The idea is that you should always double the cluster with Priam rather than adding nodes one-by-one.

Andor

Answer 8 · 2016-11-11T15:31:13.000Z

That seems like a poor reason to cripple this feature. Why have a move endpoint at all, then, if that's the thinking?

Answer 9 · 2016-11-11T19:09:32.000Z

Just to provide more context.
Why not add node instead of doubling? Doubling the cluster ensures balance of data.
Why not use v-nodes? v-nodes have its advantages but for Netflix, it has one critical flaw, token range offline. Specifically, there is a possibility of replication residing in same zones. Hence losing the zone will result in token range offline.

Answer 10 · 2016-11-11T20:23:04.000Z

I know why doubling is better. That doesn't address the issue of move bot behaving the way it should. Either implement it correctly or remove it.

That aside, in my experience using the double ring endpoint does not result in a balanced cluster. As a result I have to move nodes. I thought this was by design, but based on this discussion that doesn't seem to be the case.

Answer 11 · 2017-01-25T20:59:59.000Z

No, double-ring endpoint works properly for us (Xively), so you might have misconfigured something or you have a wrong ASG setup.
/move endpoint however should be extended with an update in SimpleDb, because currently it only forwards the request to nodetool.
It would be nice if someone could create PR on that.