Once a leader always a leader...

Hi,

Okay, sorry for raising all these issues :)

This one I'm also a little unsure of. Basically I've got the nodes communicating but I have not wired up the ReceiveAppendEntries. So what I expect to see is that they have a vote, one of them wins the vote and becomes the leader and then they start sending AppendEntries. However the other node can't see this so it should timeout and restart the voting process - which it does.

When the first node gets the first vote request, it votes no because it is already in the term. It then gets another vote request for the the next term which it then correctly votes yes to.

But here's the weird thing - after it votes yes, it goes back to sending AppendEntries. This means after a few terms, all my nodes are sending them as they all think they're the leader. Here's some of the output:

Node1 is requesting a vote on term 1
Response:  Term [1] Vote Granted [0]

Timer hit: Send append entries!

Node1 is requesting a vote on term 2
Response:  Term [1] Vote Granted [1]

Timer hit: Send append entries!

I would expect that after a Leader grants a vote, it should step down as the Leader right? I may actually have this totally wrong, and maybe this is not the behaviour I should expect.

If this is the right behaviour, then I guess we need to update:

raft/src/raft_server.c

Line 334 in f53a72e

int raft_recv_requestvote(raft_server_t* me_, int node, msg_requestvote_t* vr,

What do you think?

Cheers,

Pete

PS How do you enable the built in logging that I just noticed? :)

The Raft paper doesn't say anything about the server stepping down when it votes yes (FWIW it seems intuitive to me that it should step down). I believe Hashicorp's implementation doesn't step down either. Raft servers step down when another node has a term which is newer (eg. when receiving AppendEntries).

This means the bug is somewhere else, possibly to do with sending/receiving AppendEntries. I will try to have a look

Re: built in logging, just switch the #if 0 to 1 here:
https://github.com/willemt/raft/blob/master/src/raft_server.c#L32

In that case leave it with me for a bit. I wasn't expecting this behaviour but if it is expected, then it could well be me (again).

Please hold fire until I get back to you :)

Also I replied to your email :)

Okay, this turned out to be my fault. I assumed it would step down on voting for another node. Now that I've hooked up append_entries it's working as specified (or as unspecified depending on your point of view) in the paper :)

I haven't been paying much attention, but I saw this and wanted to try to clarify.

A server that grants a vote should be in the follower state for any normal implementation:

To grant a vote, the server must have a vote available in the given term.
When a server enters the candidate state, it votes for itself (in any normal implementation).
A server can only get to the leader state from the candidate state.
So the only way a server can have a vote available is for it to be in the follower state.

In many implementations, a candidate or leader can grant the same exact vote in the same exact term again if the same server asks again in the same term. But that's not exactly the same as granting the vote. The vote has already been granted really, it's just that reply telling the other server as much was lost, so the other server has asked again.

Thanks for the clarification @ongardie! That makes sense to me now

@ongardie , the reason I'm working on this at the moment is to get Raft working over XMPP. There's an experimental XEP for it already:

http://xmpp.org/extensions/xep-0362.html

So thanks for all your hard work :D

Cool, @pmembrey. I'm pretty sure you're the first to do Raft over XMPP. That's not something I ever expected to happen, but then again I only know XMPP for basic online chat. After reading the intro to your doc, I see how it'd make sense in some contexts, so thanks for explaining :)