zllovesuki/specter

chord/membership: incorrect successor when joining

zllovesuki opened this issue · 1 comments

        logger.go:130: 2023-01-26T07:58:28.720Z	INFO	chord/local_membership.go:55	Successfully joined Chord ring	{"node": 114622756829, "predecessor": 93693217140, "successor": 119592975162}
        logger.go:130: 2023-01-26T07:58:28.720Z	INFO	chord/local_membership.go:148	Join completed, joiner has requested to update pointers	{"node": 93693217140}
        logger.go:130: 2023-01-26T07:58:28.721Z	INFO	chord/local_tasks.go:133	FingerTable entries updated	{"node": 93693217140, "fixed": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35]}
        logger.go:130: 2023-01-26T07:58:28.721Z	INFO	chord/local_membership.go:156	Join completed, joiner has requested to release membership lock	{"node": 119592975162}
        logger.go:130: 2023-01-26T07:58:28.721Z	INFO	chord/local_membership.go:71	Joining Chord ring	{"node": 644634094637, "via": ""}
        logger.go:130: 2023-01-26T07:58:28.721Z	INFO	chord/local_membership.go:100	incoming join request	{"node": 703332983828, "joiner": 644634094637}
        !!! -> logger.go:130: 2023-01-26T07:58:28.722Z	DEBUG	chord/local_chord.go:216	skip transferring keys to predecessor because predecessor left	{"node": 703332983828, "prev": 688551194597, "new": 644634094637}
        logger.go:130: 2023-01-26T07:58:28.722Z	INFO	chord/local_tasks.go:133	FingerTable entries updated	{"node": 644634094637, "fixed": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42]}
        logger.go:130: 2023-01-26T07:58:28.722Z	INFO	chord/local_membership.go:55	Successfully joined Chord ring	{"node": 644634094637, "predecessor": 688551194597, "successor": 703332983828}
        logger.go:130: 2023-01-26T07:58:28.722Z	INFO	chord/local_tasks.go:133	FingerTable entries updated	{"node": 3930596034382, "fixed": [39]}
        logger.go:130: 2023-01-26T07:58:28.722Z	INFO	chord/local_membership.go:148	Join completed, joiner has requested to update pointers	{"node": 688551194597}
        logger.go:130: 2023-01-26T07:58:28.723Z	INFO	chord/local_chord.go:91	Discovered new predecessor via Notify	{"node": 703332983828, "previous": 644634094637, "predecessor": 688551194597}
        logger.go:130: 2023-01-26T07:58:28.723Z	INFO	chord/local_tasks.go:98	Discovered new successors via Stablize	{"node": 329708973517, "successors": [688551194597, 703332983828, 1008538930581, 1012275919287]}

Since we are always calling .Join() on nodes[0], it is possible that its finger table is not updated yet:

  • node 703332983828 has predecessor 688551194597 and successor (not important)
    • INFO chord/local_membership.go:55 Successfully joined Chord ring {"node": 703332983828, "predecessor": 688551194597, "successor": 1008538930581}
  • node 644634094637 has requested to join via nodes[0], found that 644634094637's successor to be 703332983828
    • INFO chord/local_membership.go:100 incoming join request {"node": 703332983828, "joiner": 644634094637}
  • however 644634094637's successor should really be 688551194597, thus causing this failure
    • DEBUG chord/local_chord.go:216 skip transferring keys to predecessor because predecessor left {"node": 703332983828, "prev": 688551194597, "new": 644634094637}

Should add additional check to ensure that the join request is routed to the correct successor in .RequestToJoin()