chord/membership: incorrect successor when joining
zllovesuki opened this issue · 1 comments
zllovesuki commented
logger.go:130: 2023-01-26T07:58:28.720Z INFO chord/local_membership.go:55 Successfully joined Chord ring {"node": 114622756829, "predecessor": 93693217140, "successor": 119592975162}
logger.go:130: 2023-01-26T07:58:28.720Z INFO chord/local_membership.go:148 Join completed, joiner has requested to update pointers {"node": 93693217140}
logger.go:130: 2023-01-26T07:58:28.721Z INFO chord/local_tasks.go:133 FingerTable entries updated {"node": 93693217140, "fixed": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35]}
logger.go:130: 2023-01-26T07:58:28.721Z INFO chord/local_membership.go:156 Join completed, joiner has requested to release membership lock {"node": 119592975162}
logger.go:130: 2023-01-26T07:58:28.721Z INFO chord/local_membership.go:71 Joining Chord ring {"node": 644634094637, "via": ""}
logger.go:130: 2023-01-26T07:58:28.721Z INFO chord/local_membership.go:100 incoming join request {"node": 703332983828, "joiner": 644634094637}
!!! -> logger.go:130: 2023-01-26T07:58:28.722Z DEBUG chord/local_chord.go:216 skip transferring keys to predecessor because predecessor left {"node": 703332983828, "prev": 688551194597, "new": 644634094637}
logger.go:130: 2023-01-26T07:58:28.722Z INFO chord/local_tasks.go:133 FingerTable entries updated {"node": 644634094637, "fixed": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42]}
logger.go:130: 2023-01-26T07:58:28.722Z INFO chord/local_membership.go:55 Successfully joined Chord ring {"node": 644634094637, "predecessor": 688551194597, "successor": 703332983828}
logger.go:130: 2023-01-26T07:58:28.722Z INFO chord/local_tasks.go:133 FingerTable entries updated {"node": 3930596034382, "fixed": [39]}
logger.go:130: 2023-01-26T07:58:28.722Z INFO chord/local_membership.go:148 Join completed, joiner has requested to update pointers {"node": 688551194597}
logger.go:130: 2023-01-26T07:58:28.723Z INFO chord/local_chord.go:91 Discovered new predecessor via Notify {"node": 703332983828, "previous": 644634094637, "predecessor": 688551194597}
logger.go:130: 2023-01-26T07:58:28.723Z INFO chord/local_tasks.go:98 Discovered new successors via Stablize {"node": 329708973517, "successors": [688551194597, 703332983828, 1008538930581, 1012275919287]}
zllovesuki commented
Since we are always calling .Join()
on nodes[0]
, it is possible that its finger table is not updated yet:
- node 703332983828 has predecessor 688551194597 and successor (not important)
INFO chord/local_membership.go:55 Successfully joined Chord ring {"node": 703332983828, "predecessor": 688551194597, "successor": 1008538930581}
- node 644634094637 has requested to join via
nodes[0]
, found that 644634094637's successor to be 703332983828INFO chord/local_membership.go:100 incoming join request {"node": 703332983828, "joiner": 644634094637}
- however 644634094637's successor should really be 688551194597, thus causing this failure
DEBUG chord/local_chord.go:216 skip transferring keys to predecessor because predecessor left {"node": 703332983828, "prev": 688551194597, "new": 644634094637}
Should add additional check to ensure that the join request is routed to the correct successor in .RequestToJoin()