nakamoto p2p: panicked at 'assertion failed: peer.last_asked.as_ref() != Some(&locators)'

[vincenzopalazzo]

This is the log that I see on my node

I am writing a lightning node with the ldk using nakamoto, it is not completed al all but I found a strange crash
this morning that I am leaving it there

Looks like that nakamoto crash with the following assertion

thread '<unnamed>' panicked at 'assertion failed: peer.last_asked.as_ref() != Some(&locators)', /home/vincent/.cargo/git/checkouts/nakamoto-90f2d5c288bae2ec/8744d43/p2p/src/fsm/syncmgr.rs:425:13

A full log is reported below.

2023-04-10T12:23:47.342Z WARN p2p Error receiving filter headers: ignoring message from [2001:19f0:6402:89:5400:4ff:fe4e:8adf]:18333: previous filter header does not match local tip
2023-04-10T12:23:48.113Z DEBUG p2p Received "headers" from [2001:19f0:6402:89:5400:4ff:fe4e:8adf]:18333
2023-04-10T12:23:48.113Z DEBUG nakamoto_p2p::fsm::syncmgr [sync] Received 1 block header(s) from [2001:19f0:6402:89:5400:4ff:fe4e:8adf]:18333
2023-04-10T12:23:48.113Z INFO p2p Block 000000000000e9ea85f16406a8f1f641ffc0c9623c4b241e26aa43d2053ba8f9 connected at height 2428316
2023-04-10T12:23:48.113Z INFO p2p Headers synced up to height 2428316 with hash 000000000000e9ea85f16406a8f1f641ffc0c9623c4b241e26aa43d2053ba8f9
2023-04-10T12:23:48.114Z INFO p2p Headers synced up to height 2428316 with hash 000000000000e9ea85f16406a8f1f641ffc0c9623c4b241e26aa43d2053ba8f9
2023-04-10T12:23:48.114Z INFO p2p Filter header chain is out of sync by 618 header(s) (2427698 to 2428316)
2023-04-10T12:23:48.114Z DEBUG p2p Sending "getcfheaders" to [2001:19f0:6402:89:5400:4ff:fe4e:8adf]:18333
2023-04-10T12:23:48.114Z INFO p2p Syncing filter headers with [2001:19f0:6402:89:5400:4ff:fe4e:8adf]:18333 from height 2427699 to 2428316 (block hash 000000000000e9ea85f16406a8f1f641ffc0c9623c4b241e26aa43d2053ba8f9)
2023-04-10T12:23:48.114Z DEBUG p2p Received "headers" from 45.76.236.69:18333
2023-04-10T12:23:48.114Z DEBUG nakamoto_p2p::fsm::syncmgr [sync] Received 1 block header(s) from 45.76.236.69:18333
2023-04-10T12:23:48.114Z DEBUG p2p Received "headers" from 35.212.255.229:18333
2023-04-10T12:23:48.114Z DEBUG nakamoto_p2p::fsm::syncmgr [sync] Received 1 block header(s) from 35.212.255.229:18333
2023-04-10T12:23:48.114Z DEBUG p2p Received "inv" from 35.214.213.127:18333
2023-04-10T12:23:48.114Z INFO p2p 35.214.213.127:18333: Discovered new block: 000000008a5eef11c068509444bc7d7e89fe366726b1ade9f6f821f0cd29d6b1
2023-04-10T12:23:48.114Z DEBUG p2p Sending "getheaders" to 35.214.213.127:18333
2023-04-10T12:23:48.115Z DEBUG nakamoto_client::service Write RawNetworkMessage { magic: 118034699, payload: GetCFHeaders(GetCFHeaders { filter_type: 0, start_height: 2427699, stop_hash: 000000000000e9ea85f16406a8f1f641ffc0c9623c4b241e26aa43d2053ba8f9 }) } to 2001:19f0:6402:89:5400:4ff:fe4e:8adf
2023-04-10T12:23:48.115Z DEBUG nakamoto_client::service Write RawNetworkMessage { magic: 118034699, payload: GetHeaders(GetHeadersMessage { version: 70016, locator_hashes: [000000000000e9ea85f16406a8f1f641ffc0c9623c4b241e26aa43d2053ba8f9, 00000000000016802f5f22dfd3109c6282f600845e950a54372ed4c3399188c9, 000000000000000e05885585fee6019dd05b2c5db17d61d2c7eca4ea78020155, 0000000000001b2c8529901836c39a450fc31bf61a8fb9e30811ff58f79d47bf, 00000000000000119a17f9acf99d53720afc97a28ddca3550329cfac335df8b1, 00000000000000113593b5b8992e4276c609ab7e3d12b942a555ca1079066bcd, 0000000000000019609321bcf6323650fe737c3bc9c4011650cda4fa0d2499c8, 00000000000069022b2de75a9457275cde3c508d54d961e084e0d237c35e71b5, 00000000eec0ec264c1a6e648bbcf318d82e076039692abf243043e7e2b9111d, 000000000000000aef5999ee05eb018888766dcd63f91d9121c3a717ab59dda7, 0000000000006f6db17813af4d0cd0201d421baa43db7bd613a0543e33dd6514, 00000000000000132f067bf278c24063822e5e122ecad7986f7c072645894890, 0000000000000009a775f2b20eb909752d3d57eae0fd5d9f0ed521c49ef144bf, 000000000000002613a8c8fab9051c0e6ac07c9eacb9cd5fa1ed436f617e4478, 000000000000000f3c35e62957d7495744ddf5985c781a279163bb1d80eefaaf, 00000000000000162f2a57878b6a7eb7e908df78c46831229a243866f1d9160f, 0000000000000005b11efef006719f1bf1fd5b49f67a928d9ef32d376b5d7357, 0000000000002205f81e66eaf6be01257d5e656e76a3ee428b22bcafd13f1915, 00000000cb1d7356d7658874263ca095429a98ef9a05c65c5641af7e3b828b24, 00000000000000185c13cb1113b24eafebdcfe21876db1805b32cfe9e538459e, 00000000000000007b5dfdb35e9400b69abe9b27f42b8b7516d9e49c70638021, 000000000000002ee3cd3b70485ce9f4df68f27faf407ae2a2fdf09fcfe94d38, 0000000000009e01086a2ff1dafb169ab8fcc12ee7998a8a371421b796007e8a, 000000000000a57c51ffa2de52b238154578b51e9ee067f59b96154c887d7d6f, 0000000000013ebf77921352a1350efa8027d9b6f68c7f0bb0b56ab97f94f826, 000000000000c04b88528605a8b18606913080e449c528924ef2c9deddc5598f, 00000000483d88dc015e55d833403af14c074e0bb8dd159afc29596b0683750f, 00000000000003158e02031c49590d41ed6672d5d6ae01d7b8000a5919efc563, 00000000065cb17eaac6c309ef037156aeb71aa50ff538a1775b38a04aa50863], stop_hash: 000000008a5eef11c068509444bc7d7e89fe366726b1ade9f6f821f0cd29d6b1 }) } to 35.214.213.127
2023-04-10T12:23:48.137Z DEBUG p2p Received "headers" from 66.183.0.205:18333
2023-04-10T12:23:48.137Z DEBUG nakamoto_p2p::fsm::syncmgr [sync] Received 1 block header(s) from 66.183.0.205:18333
2023-04-10T12:23:48.156Z DEBUG p2p Received "headers" from 35.214.213.127:18333
2023-04-10T12:23:48.156Z DEBUG nakamoto_p2p::fsm::syncmgr [sync] Received 1 block header(s) from 35.214.213.127:18333
2023-04-10T12:23:48.156Z INFO p2p Block 000000000000e9ea85f16406a8f1f641ffc0c9623c4b241e26aa43d2053ba8f9 disconnected at height 2428316
2023-04-10T12:23:48.156Z INFO p2p Block 000000008a5eef11c068509444bc7d7e89fe366726b1ade9f6f821f0cd29d6b1 connected at height 2428316
2023-04-10T12:23:48.156Z INFO p2p Headers synced up to height 2428316 with hash 000000008a5eef11c068509444bc7d7e89fe366726b1ade9f6f821f0cd29d6b1
2023-04-10T12:23:48.156Z INFO p2p Headers synced up to height 2428316 with hash 000000008a5eef11c068509444bc7d7e89fe366726b1ade9f6f821f0cd29d6b1
2023-04-10T12:23:48.157Z INFO p2p Filter header chain is out of sync by 618 header(s) (2427698 to 2428316)
2023-04-10T12:23:48.157Z DEBUG p2p Sending "getcfheaders" to 66.183.0.205:18333
2023-04-10T12:23:48.157Z INFO p2p Syncing filter headers with 66.183.0.205:18333 from height 2427699 to 2428316 (block hash 000000008a5eef11c068509444bc7d7e89fe366726b1ade9f6f821f0cd29d6b1)
2023-04-10T12:23:48.157Z DEBUG nakamoto_client::service Write RawNetworkMessage { magic: 118034699, payload: GetCFHeaders(GetCFHeaders { filter_type: 0, start_height: 2427699, stop_hash: 000000008a5eef11c068509444bc7d7e89fe366726b1ade9f6f821f0cd29d6b1 }) } to 66.183.0.205
2023-04-10T12:23:48.233Z DEBUG p2p Received "headers" from 34.210.154.40:18333
2023-04-10T12:23:48.233Z DEBUG nakamoto_p2p::fsm::syncmgr [sync] Received 1 block header(s) from 34.210.154.40:18333
2023-04-10T12:23:48.235Z DEBUG p2p Sending "getheaders" to 34.210.154.40:18333
2023-04-10T12:23:48.235Z DEBUG nakamoto_client::service Write RawNetworkMessage { magic: 118034699, payload: GetHeaders(GetHeadersMessage { version: 70016, locator_hashes: [000000008a5eef11c068509444bc7d7e89fe366726b1ade9f6f821f0cd29d6b1, 00000000000016802f5f22dfd3109c6282f600845e950a54372ed4c3399188c9, 000000000000000e05885585fee6019dd05b2c5db17d61d2c7eca4ea78020155, 0000000000001b2c8529901836c39a450fc31bf61a8fb9e30811ff58f79d47bf, 00000000000000119a17f9acf99d53720afc97a28ddca3550329cfac335df8b1, 00000000000000113593b5b8992e4276c609ab7e3d12b942a555ca1079066bcd, 0000000000000019609321bcf6323650fe737c3bc9c4011650cda4fa0d2499c8, 00000000000069022b2de75a9457275cde3c508d54d961e084e0d237c35e71b5, 00000000eec0ec264c1a6e648bbcf318d82e076039692abf243043e7e2b9111d, 000000000000000aef5999ee05eb018888766dcd63f91d9121c3a717ab59dda7, 0000000000006f6db17813af4d0cd0201d421baa43db7bd613a0543e33dd6514, 00000000000000132f067bf278c24063822e5e122ecad7986f7c072645894890, 0000000000000009a775f2b20eb909752d3d57eae0fd5d9f0ed521c49ef144bf, 000000000000002613a8c8fab9051c0e6ac07c9eacb9cd5fa1ed436f617e4478, 000000000000000f3c35e62957d7495744ddf5985c781a279163bb1d80eefaaf, 00000000000000162f2a57878b6a7eb7e908df78c46831229a243866f1d9160f, 0000000000000005b11efef006719f1bf1fd5b49f67a928d9ef32d376b5d7357, 0000000000002205f81e66eaf6be01257d5e656e76a3ee428b22bcafd13f1915, 00000000cb1d7356d7658874263ca095429a98ef9a05c65c5641af7e3b828b24, 00000000000000185c13cb1113b24eafebdcfe21876db1805b32cfe9e538459e, 00000000000000007b5dfdb35e9400b69abe9b27f42b8b7516d9e49c70638021, 000000000000002ee3cd3b70485ce9f4df68f27faf407ae2a2fdf09fcfe94d38, 0000000000009e01086a2ff1dafb169ab8fcc12ee7998a8a371421b796007e8a, 000000000000a57c51ffa2de52b238154578b51e9ee067f59b96154c887d7d6f, 0000000000013ebf77921352a1350efa8027d9b6f68c7f0bb0b56ab97f94f826, 000000000000c04b88528605a8b18606913080e449c528924ef2c9deddc5598f, 00000000483d88dc015e55d833403af14c074e0bb8dd159afc29596b0683750f, 00000000000003158e02031c49590d41ed6672d5d6ae01d7b8000a5919efc563, 00000000065cb17eaac6c309ef037156aeb71aa50ff538a1775b38a04aa50863], stop_hash: 000000000000e9ea85f16406a8f1f641ffc0c9623c4b241e26aa43d2053ba8f9 }) } to 34.210.154.40
2023-04-10T12:23:48.241Z DEBUG p2p Received "inv" from 203.132.94.196:18333
2023-04-10T12:23:48.300Z DEBUG p2p Received "inv" from 110.141.252.80:18333
2023-04-10T12:23:48.313Z DEBUG p2p Received "cfheaders" from [2001:19f0:6402:89:5400:4ff:fe4e:8adf]:18333
2023-04-10T12:23:48.313Z DEBUG spv Received 618 filter header(s) from [2001:19f0:6402:89:5400:4ff:fe4e:8adf]:18333
2023-04-10T12:23:48.313Z DEBUG p2p Disconnecting from [2001:19f0:6402:89:5400:4ff:fe4e:8adf]:18333: peer misbehaving: unknown `cfheaders` stop hash
2023-04-10T12:23:48.313Z INFO p2p Disconnected from [2001:19f0:6402:89:5400:4ff:fe4e:8adf]:18333 (peer misbehaving: unknown `cfheaders` stop hash)
2023-04-10T12:23:48.314Z INFO p2p Connecting to peer 54.215.91.44:18333 from source `84.27.211.215:18333` with ServiceFlags(NETWORK|WITNESS|COMPACT_FILTERS|NETWORK_LIMITED)
2023-04-10T12:23:48.350Z INFO p2p Disconnected from 66.183.0.205:18333 (connection reset)
2023-04-10T12:23:48.351Z INFO p2p Connecting to peer 16.162.119.13:18333 from source `3.71.96.36:18333` with ServiceFlags(NETWORK|WITNESS|COMPACT_FILTERS|NETWORK_LIMITED)
2023-04-10T12:23:48.433Z INFO p2p Filter header chain is out of sync by 618 header(s) (2427698 to 2428316)
2023-04-10T12:23:48.437Z DEBUG p2p Received "headers" from 34.210.154.40:18333
2023-04-10T12:23:48.438Z DEBUG nakamoto_p2p::fsm::syncmgr [sync] Received 1 block header(s) from 34.210.154.40:18333
thread '<unnamed>' panicked at 'assertion failed: peer.last_asked.as_ref() != Some(&locators)', /home/vincent/.cargo/git/checkouts/nakamoto-90f2d5c288bae2ec/8744d43/p2p/src/fsm/syncmgr.rs:425:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2023-04-10T12:24:04.782Z DEBUG lampod::utils::logger -1 TRACE [lightning_background_processor:597] Calling PeerManager's timer_tick_occurred

2023-04-10T12:24:06.086Z DEBUG lampod::utils::logger -1 TRACE [lightning_background_processor:597] Persisting scorer

2023-04-10T12:24:06.086Z ERROR lampod::utils::logger -1 ERROR [lightning_background_processor:597] Error: Failed to persist scorer, check your disk and permissions File exists (os error 17)

2023-04-10T12:24:15.615Z DEBUG lampod::utils::logger -1 TRACE [lightning_background_processor:597] Calling ChannelManager's timer_tick_occurred

2023-04-10T12:24:15.615Z ERROR lampo_nakamoto Err(Disconnected)
thread '<unnamed>' panicked at 'Err(Disconnected)', lampo-nakamoto/src/lib.rs:107:13
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Any { .. }', /home/vincent/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lightning-background-processor-0.0.114/src/lib.rs:641:43

In addition, I am running a custom version of Nakamoto where I join different features proposed with by me with a PR, the branch that I am running is https://github.com/vincenzopalazzo/nakamoto/tree/macros/client_model-fixes

[vincenzopalazzo]

This is strange, and I am thinking that can be caused by two Nakamoto instances that are running on the same machine.

Looking into it and trying to reproduce it