Collator stopped producing blocks

Question

Collator stopped producing blocks

Closed this issue 2 years ago · 9 comments

Collator: bifrost 0.9.43-b767dc6-x86_64-linux-gnu
Ubuntu 20.04 LTS
VPS 8 core, RAM 30GB, 400 GB NVMe disk

Jun 03 09:22:48 mn12 bifrost[2386398]: 2022-06-03 09:22:48 [Parachain🌈] Starting collation. relay_parent=0xba63063d76632501e224647ea6f1b3ba1883dd9c25c3620b853522984a921506 at=0xc16516512a5ecb64c27afe2c4e9dd49348bdce75dcddb8fab5eac7a53fb738fb
Jun 03 09:22:48 mn12 bifrost[2386398]: 2022-06-03 09:22:48 [Parachain🌈] ✨ Imported #1849598 (0xc070…757b)
Jun 03 09:22:49 mn12 bifrost[2386398]: 2022-06-03 09:22:49 [Relaychain] 💤 Idle (46 peers), best: #12964450 (0xba63…1506), finalized #12964445 (0xc612…82b1), ⬇ 139.2kiB/s ⬆ 269.7kiB/s
**Jun 03 09:22:51 mn12 bifrost[2386398]: 2022-06-03 09:22:51 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\x97\x92.2a\xa3*E\x17h3\xc8\xed\xb7\xf0e\"\xffS\xb5\xffj\xa8\x8e\xf7\xd0)\x0c\x94\xbfB\xe8"), success: [], quorum: 20 })
Jun 03 09:22:51 mn12 bifrost[2386398]: 2022-06-03 09:22:51 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\xdcD\xe1\x8c\xcb7\x08\xd0\x85\xe6U\xcfY\xbe\xb2\x11\xb9[+\xa4\xd0u}\xfa\xff\x98C)\xf5|W\xb9"), success: [], quorum: 20 })**
Jun 03 09:22:53 mn12 bifrost[2386398]: 2022-06-03 09:22:53 [Parachain🌈] 💤 Idle (58 peers), best: #1849597 (0xc165…38fb), finalized #1849595 (0x8a6b…e1bc), ⬇ 6.9kiB/s ⬆ 60.8kiB/s
Jun 03 09:22:54 mn12 bifrost[2386398]: 2022-06-03 09:22:54 [Relaychain] 💤 Idle (46 peers), best: #12964450 (0xba63…1506), finalized #12964447 (0xad18…0ca3), ⬇ 866.5kiB/s ⬆ 884.6kiB/s
Jun 03 09:22:54 mn12 bifrost[2386398]: 2022-06-03 09:22:54 [Relaychain] ✨ Imported #12964451 (0xf68f…1cbb)
Jun 03 09:22:58 mn12 bifrost[2386398]: 2022-06-03 09:22:58 [Parachain🌈] 💤 Idle (58 peers), best: #1849597 (0xc165…38fb), finalized #1849595 (0x8a6b…e1bc), ⬇ 0.9kiB/s ⬆ 1.5kiB/s
Jun 03 09:22:59 mn12 bifrost[2386398]: 2022-06-03 09:22:59 [Relaychain] 💤 Idle (46 peers), best: #12964451 (0xf68f…1cbb), finalized #12964448 (0x9338…7a00), ⬇ 877.9kiB/s ⬆ 873.2kiB/s
Jun 03 09:23:00 mn12 bifrost[2386398]: 2022-06-03 09:23:00 [Relaychain] ✨ Imported #12964452 (0x58a7…dea9)
Jun 03 09:23:00 mn12 bifrost[2386398]: 2022-06-03 09:23:00 [Parachain🌈] Starting collation. relay_parent=0x58a76be59dd40036e5084051a93d727aa2833e8bd47fe0b1021ae546d87cdea9 at=0xc07036dee0fd4b40425ae5daedf2b80829d482dac5ea36ca17c85934eff8757b
Jun 03 09:23:01 mn12 bifrost[2386398]: 2022-06-03 09:23:01 [Parachain🌈] ✨ Imported #1849599 (0x2b98…6461)
Jun 03 09:23:03 mn12 bifrost[2386398]: 2022-06-03 09:23:03 [Parachain🌈] 💤 Idle (58 peers), best: #1849598 (0xc070…757b), finalized #1849596 (0x9858…69ee), ⬇ 36.3kiB/s ⬆ 40.4kiB/s
Jun 03 09:23:04 mn12 bifrost[2386398]: 2022-06-03 09:23:04 [Relaychain] 💤 Idle (47 peers), best: #12964452 (0x58a7…dea9), finalized #12964448 (0x9338…7a00), ⬇ 180.7kiB/s ⬆ 221.9kiB/s
Jun 03 09:23:06 mn12 bifrost[2386398]: 2022-06-03 09:23:06 [Relaychain] ✨ Imported #12964453 (0xb5ff…c628)
Jun 03 09:23:08 mn12 bifrost[2386398]: 2022-06-03 09:23:08 [Parachain🌈] 💤 Idle (58 peers), best: #1849598 (0xc070…757b), finalized #1849596 (0x9858…69ee), ⬇ 0.8kiB/s ⬆ 0.8kiB/s
Jun 03 09:23:09 mn12 bifrost[2386398]: 2022-06-03 09:23:09 [Relaychain] 💤 Idle (46 peers), best: #12964453 (0xb5ff…c628), finalized #12964449 (0x7816…02c0), ⬇ 631.3kiB/s ⬆ 718.9kiB/s
Jun 03 09:23:12 mn12 bifrost[2386398]: 2022-06-03 09:23:12 [Relaychain] ✨ Imported #12964454 (0xa33a…297d)
Jun 03 09:23:12 mn12 bifrost[2386398]: 2022-06-03 09:23:12 [Parachain🌈] Starting collation. relay_parent=0xa33aabcfb6660545fa7dda7d67c725e02358e9a1db0d0e14ea1292f48d6e297d at=0x2b98d876533a7c67ac8c699282b128cd9c2dd378e2afb16a0304ee7d42b36461
Jun 03 09:23:12 mn12 bifrost[2386398]: 2022-06-03 09:23:12 [Parachain🌈] ✨ Imported #1849600 (0xb4ad…8357)
Jun 03 09:23:13 mn12 bifrost[2386398]: 2022-06-03 09:23:13 [Parachain🌈] 💤 Idle (57 peers), best: #1849599 (0x2b98…6461), finalized #1849596 (0x9858…69ee), ⬇ 25.8kiB/s ⬆ 39.3kiB/s
Jun 03 09:23:14 mn12 bifrost[2386398]: 2022-06-03 09:23:14 [Relaychain] 💤 Idle (46 peers), best: #12964454 (0xa33a…297d), finalized #12964449 (0x7816…02c0), ⬇ 109.1kiB/s ⬆ 217.7kiB/s
Jun 03 09:23:18 mn12 bifrost[2386398]: 2022-06-03 09:23:18 [Parachain🌈] 💤 Idle (59 peers), best: #1849599 (0x2b98…6461), finalized #1849597 (0xc165…38fb), ⬇ 1.3kiB/s ⬆ 4.7kiB/s
Jun 03 09:23:18 mn12 bifrost[2386398]: 2022-06-03 09:23:18 [Relaychain] ✨ Imported #12964455 (0xa229…a1c7)
Jun 03 09:23:19 mn12 bifrost[2386398]: 2022-06-03 09:23:19 [Relaychain] 💤 Idle (45 peers), best: #12964455 (0xa229…a1c7), finalized #12964450 (0xba63…1506), ⬇ 392.3kiB/s ⬆ 867.0kiB/s
Jun 03 09:23:23 mn12 bifrost[2386398]: 2022-06-03 09:23:23 [Parachain🌈] 💤 Idle (59 peers), best: #1849599 (0x2b98…6461), finalized #1849598 (0xc070…757b), ⬇ 0.8kiB/s ⬆ 0.8kiB/s
Jun 03 09:23:24 mn12 bifrost[2386398]: 2022-06-03 09:23:24 [Relaychain] ✨ Imported #12964456 (0x6729…9356)
Jun 03 09:23:24 mn12 bifrost[2386398]: 2022-06-03 09:23:24 [Parachain🌈] Starting collation. relay_parent=0x6729788f2dee3c7cb824ec33ba02898d13dbff0f72a793176164b1156e0f9356 at=0xb4ad33defd538344484a43ca028f24ec068c0310507968c1815db1a352d78357
Jun 03 09:23:24 mn12 bifrost[2386398]: 2022-06-03 09:23:24 [Relaychain] ♻️  Reorg on #12964456,0x6729…9356 to #12964456,0xe6e2…78be, common ancestor #12964455,0xa229…a1c7
Jun 03 09:23:24 mn12 bifrost[2386398]: 2022-06-03 09:23:24 [Relaychain] ✨ Imported #12964456 (0xe6e2…78be)
Jun 03 09:23:24 mn12 bifrost[2386398]: 2022-06-03 09:23:24 [Parachain🌈] Starting collation. relay_parent=0xe6e2c843085cde89734f2bd1abe84c5225786d818be3a306911fc3f4854278be at=0xb4ad33defd538344484a43ca028f24ec068c0310507968c1815db1a352d78357
Jun 03 09:23:24 mn12 bifrost[2386398]: 2022-06-03 09:23:24 [Parachain🌈] ✨ Imported #1849601 (0xcfa8…e0fa)
Jun 03 09:23:24 mn12 bifrost[2386398]: 2022-06-03 09:23:24 [Relaychain] ✨ Imported #12964456 (0xda4f…8001)
Jun 03 09:23:24 mn12 bifrost[2386398]: 2022-06-03 09:23:24 [Parachain🌈] Starting collation. relay_parent=0xda4fbaec8405fd32ae987949ca99be1bab6cde2152c5328253a6a73b91ff8001 at=0xb4ad33defd538344484a43ca028f24ec068c0310507968c1815db1a352d78357
Jun 03 09:23:24 mn12 bifrost[2386398]: 2022-06-03 09:23:24 [Relaychain] 💤 Idle (44 peers), best: #12964456 (0xe6e2…78be), finalized #12964452 (0x58a7…dea9), ⬇ 498.2kiB/s ⬆ 702.4kiB/s
Jun 03 09:23:25 mn12 bifrost[2386398]: 2022-06-03 09:23:25 [Parachain🌈] ✨ Imported #1849601 (0x193c…226c)
Jun 03 09:23:25 mn12 bifrost[2386398]: 2022-06-03 09:23:25 [Parachain🌈] ✨ Imported #1849601 (0xc724…7e4c)
Jun 03 09:23:26 mn12 bifrost[2386398]: 2022-06-03 09:23:26 [Parachain🌈] ✨ Imported #1849601 (0x4033…7c2b)
Jun 03 09:23:26 mn12 bifrost[2386398]: 2022-06-03 09:23:26 [Parachain🌈] ✨ Imported #1849601 (0x561c…d46d)
Jun 03 09:23:26 mn12 bifrost[2386398]: 2022-06-03 09:23:26 [Parachain🌈] ✨ Imported #1849601 (0x0499…e61f)
Jun 03 09:23:28 mn12 bifrost[2386398]: 2022-06-03 09:23:28 [Parachain🌈] 💤 Idle (57 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 169.1kiB/s ⬆ 171.0kiB/s
Jun 03 09:23:33 mn12 bifrost[2386398]: 2022-06-03 09:23:33 [Parachain🌈] 💤 Idle (59 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 1.3kiB/s ⬆ 1.5kiB/s
Jun 03 09:23:38 mn12 bifrost[2386398]: 2022-06-03 09:23:38 [Parachain🌈] 💤 Idle (59 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 1.1kiB/s ⬆ 1.2kiB/s
Jun 03 09:23:43 mn12 bifrost[2386398]: 2022-06-03 09:23:43 [Parachain🌈] 💤 Idle (59 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 8.6kiB/s ⬆ 1.2kiB/s
Jun 03 09:23:48 mn12 bifrost[2386398]: 2022-06-03 09:23:48 [Parachain🌈] 💤 Idle (59 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 0.8kiB/s ⬆ 1.0kiB/s
Jun 03 09:23:53 mn12 bifrost[2386398]: 2022-06-03 09:23:53 [Parachain🌈] 💤 Idle (59 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 0.7kiB/s ⬆ 0.7kiB/s
Jun 03 09:23:58 mn12 bifrost[2386398]: 2022-06-03 09:23:58 [Parachain🌈] 💤 Idle (59 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 0.6kiB/s ⬆ 0.9kiB/s
Jun 03 09:24:03 mn12 bifrost[2386398]: 2022-06-03 09:24:03 [Parachain🌈] 💤 Idle (59 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 8.4kiB/s ⬆ 1.0kiB/s
Jun 03 09:24:07 mn12 bifrost[2386398]: 2022-06-03 09:24:07 [Parachain🌈] ✨ Imported #1849602 (0x5ece…de67)
Jun 03 09:24:08 mn12 bifrost[2386398]: 2022-06-03 09:24:08 [Parachain🌈] 💤 Idle (57 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 4.4kiB/s ⬆ 1.2kiB/s
Jun 03 09:24:13 mn12 bifrost[2386398]: 2022-06-03 09:24:13 [Parachain🌈] 💤 Idle (59 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 8.4kiB/s ⬆ 0.9kiB/s
Jun 03 09:24:18 mn12 bifrost[2386398]: 2022-06-03 09:24:18 [Parachain🌈] 💤 Idle (59 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 1.3kiB/s ⬆ 1.5kiB/s
Jun 03 09:24:23 mn12 bifrost[2386398]: 2022-06-03 09:24:23 [Parachain🌈] 💤 Idle (59 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 0.7kiB/s ⬆ 0.7kiB/s
Jun 03 09:24:28 mn12 bifrost[2386398]: 2022-06-03 09:24:28 [Parachain🌈] 💤 Idle (59 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 15.4kiB/s ⬆ 0.5kiB/s
Jun 03 09:24:33 mn12 bifrost[2386398]: 2022-06-03 09:24:33 [Parachain🌈] 💤 Idle (59 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 8.8kiB/s ⬆ 0.7kiB/s
Jun 03 09:24:38 mn12 bifrost[2386398]: 2022-06-03 09:24:38 [Parachain🌈] 💤 Idle (59 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 0.8kiB/s ⬆ 0.9kiB/s
Jun 03 09:24:43 mn12 bifrost[2386398]: 2022-06-03 09:24:43 [Parachain🌈] 💤 Idle (59 peers), best: #1849600 (0xb4ad…8357), finalized #1849598 (0xc070…757b), ⬇ 0.4kiB/s ⬆ 0.4kiB/s

Answer 1 · 2022-06-04T08:48:55.000Z

Last night:

Jun 03 21:39:10 mn12 bifrost[2426279]: 2022-06-03 21:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"x\xba\xb6\xa3@\xf8j<P\x85\xe2I\x84J\x11\xfb\xd3\x96Q\xa6\xe2l\xdc%\xa5)\xb2)r\xb4\x84\xea"), success: [                                                 ], quorum: 20 })
Jun 03 21:39:10 mn12 bifrost[2426279]: 2022-06-03 21:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\xc8\x08M\xd0\x1a\xb7\x9c\xbe\xf8\xc09\x1f\x81\x86\xfb\x0e\x14\x0f\xec\x82F\x8d*\xc3\xe4m\xb3\x88\xaf\x                                                 8a=\x13"), success: [], quorum: 20 })
Jun 03 21:39:10 mn12 bifrost[2426279]: 2022-06-03 21:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\xe3\xc8\xabjqS!7I\x05\xa8E\x02\xbdx\xe7^\x1d\xc1\xdc\xbc\xb2\xb9|\xeem`\xbe\xc9\x7f\x04\x11"), success                                                 : [], quorum: 20 })
Jun 04 00:39:10 mn12 bifrost[2426279]: 2022-06-04 00:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"1Oe=M;\xf8\x8fi\x81\0\x8b!=\xda\x9a\x11]\xf0[v\xc7\xf1\xac\x1c\xcf\xd5\xff\xed\xd2\xdd]"), success: [],                                                  quorum: 20 })
Jun 04 00:39:10 mn12 bifrost[2426279]: 2022-06-04 00:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\xe3\xc8\xabjqS!7I\x05\xa8E\x02\xbdx\xe7^\x1d\xc1\xdc\xbc\xb2\xb9|\xeem`\xbe\xc9\x7f\x04\x11"), success                                                 : [], quorum: 20 })
Jun 04 00:39:10 mn12 bifrost[2426279]: 2022-06-04 00:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"R\xce~M\x9a\xedP\xa1\x17\xd7\x18\xb0\xe9V\xedr\xac\x80\xdd\xca\x93\xd8\xd6\xd8C'=\x7ft\xa3\x97\xba"), s                                                 uccess: [], quorum: 20 })
Jun 04 00:39:10 mn12 bifrost[2426279]: 2022-06-04 00:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\x15\xc8\x13\xd8\x99\xa5\xcfx\x120B98^X\x80\x90\xb8\xc0\xd2\x1c\x01!\xbb/Z:T\xffF*K"), success: [], quo                                                 rum: 20 })
Jun 04 01:39:10 mn12 bifrost[2426279]: 2022-06-04 01:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\x08\x84:\xcf\xdf\x18y\xcd\t\xef{\xa8\xed\x06w\xa1I\xb6\xab\xbe\x03\xfb\xcf\x1a&\x93\tCZ_\xe9\xe5"), su                                                 ccess: [], quorum: 20 })
Jun 04 03:39:10 mn12 bifrost[2426279]: 2022-06-04 03:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\x08\x84:\xcf\xdf\x18y\xcd\t\xef{\xa8\xed\x06w\xa1I\xb6\xab\xbe\x03\xfb\xcf\x1a&\x93\tCZ_\xe9\xe5"), su                                                 ccess: [], quorum: 20 })
Jun 04 05:39:10 mn12 bifrost[2426279]: 2022-06-04 05:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\x08\x84:\xcf\xdf\x18y\xcd\t\xef{\xa8\xed\x06w\xa1I\xb6\xab\xbe\x03\xfb\xcf\x1a&\x93\tCZ_\xe9\xe5"), su                                                 ccess: [], quorum: 20 })
Jun 04 06:39:10 mn12 bifrost[2426279]: 2022-06-04 06:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\x15\xc8\x13\xd8\x99\xa5\xcfx\x120B98^X\x80\x90\xb8\xc0\xd2\x1c\x01!\xbb/Z:T\xffF*K"), success: [], quo                                                 rum: 20 })
Jun 04 09:39:10 mn12 bifrost[2426279]: 2022-06-04 09:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"R\xce~M\x9a\xedP\xa1\x17\xd7\x18\xb0\xe9V\xedr\xac\x80\xdd\xca\x93\xd8\xd6\xd8C'=\x7ft\xa3\x97\xba"), s                                                 uccess: [], quorum: 20 })
Jun 04 09:39:10 mn12 bifrost[2426279]: 2022-06-04 09:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\xa0\xb4\x15\xb9/\xe3\x82\xde\xf5\xac.g\x1d \t\xe5~4\x10I\x1b\xc3\xda\xc4\x94\x95 nQ\x0f\xe2\xd1"), suc                                                 cess: [], quorum: 20 })
Jun 04 09:39:10 mn12 bifrost[2426279]: 2022-06-04 09:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\xe3\xc8\xabjqS!7I\x05\xa8E\x02\xbdx\xe7^\x1d\xc1\xdc\xbc\xb2\xb9|\xeem`\xbe\xc9\x7f\x04\x11"), success                                                 : [], quorum: 20 })
Jun 04 09:39:10 mn12 bifrost[2426279]: 2022-06-04 09:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\xc6)\xab\x13M\xb7\xce\x84:\xd1g%\x85\xb2\xb7\xcdg6\xdd\x99\xb1\0\x9e\xfb[h[\xc63\x02\xddG"), success:                                                  [], quorum: 20 })
Jun 04 10:39:10 mn12 bifrost[2426279]: 2022-06-04 10:39:10 [Relaychain] Locating closest peers for replication failed: Err(Timeout { key: Key(b"\xba\xac\xca\x92\x18\xecQ'\x15\x0b\xb1\xd1/\xd7\x12a\x08\x7f\xc4\xa8\xcf\xecBi}\xf5\xd1\x97]\x98\xeaO")                                                 , success: [], quorum: 20 })

Answer 2 · 2022-06-06T07:01:56.000Z

Hi, the issue seems to be related to paritytech/polkadot#856 and moonbeam-foundation/moonbeam#1410

Please increase net.core.somaxconn and have try.

Answer 3 · 2022-06-07T14:36:59.000Z

Hi, I've increased:
net.core.somaxconn from 4096 to 8192
net.core.netdev_max_backlog from 1000 to 4096

Still seeing "Locating closest peers for replication failed" in logs. It's always happening at xx:39:10 hour:
Jun 07 01:39:10
Jun 07 02:39:10
Jun 07 09:39:10
Jun 07 03:39:10
Jun 07 06:39:10
Jun 07 09:39:10
Jun 07 12:39:10
Jun 07 13:39:10
Jun 07 14:39:10

Answer 4 · 2022-06-10T02:22:41.000Z

Hi @wwwAndries Could you check the max open files limit with the help of this link https://unix.stackexchange.com/questions/8945/how-can-i-increase-open-files-limit-for-all-processes

And Did you deploy the node with systemd? Then you should also check this. https://unix.stackexchange.com/a/375841

Also you need to monitor your tcp connections and the number of file_descriptors. That may help to figure the error

Answer 5 · 2022-06-10T12:20:31.000Z

Hello Bob, I run node with sysytemd. In my vi /etc/sysctl.conf I’ve this: kernel.panic = 10 vm.swappiness=1 vm.vfs_cache_pressure=100 fs.inotify.max_user_watches=20480 fs.file-max = 65536 kernel.sched_min_granularity_ns = 10000000 vm.dirty_ratio = 10 vm.dirty_background_ratio = 3 kernel.sched_migration_cost_ns = 5000000 net.core.busy_read = 50 net.core.busy_poll = 50 net.ipv4.tcp_fastopen' = 3 kernel.numa_balancing' = '' transparent_hugepages' = 'never' # Increase number of incoming connections net.core.somaxconn = 8192 # Increase number of incoming connections backlog net.core.netdev_max_backlog = 4096 In .service file: LimitNOFILE=16384 In /etc/security/limits.conf I’ve now added: bifrost soft nofile 32768 bifrost hard nofile 32768

Answer 6 · 2022-06-13T10:21:08.000Z

@wwwAndries Hi Andries, are you a validator node? what is your validator address?

Answer 7 · 2022-06-13T11:12:25.000Z

Stake.Works
cjt6kJGXdsJEAm7NqKPoDbNYktFixXaXyHr4uKvUnPjQFh3

Despite all changes I still have these errors:
[Relaychain] Locating closest peers for replication failed: Err(Timeout ...

Answer 8 · 2022-06-15T15:47:40.000Z

Please also provide your bifrost cli params to use.

Answer 9 · 2022-06-15T15:54:25.000Z

ExecStart=/usr/local/bin/bifrost/bifrost
--collator
--base-path ...
--name 'StakeWorks | CVN1201 | Bifrost'
--chain bifrost-kusama.json
--rpc-cors all
--telemetry-url 'wss://telemetry.polkadot.io/submit/ 0'
--in-peers 30
--out-peers 30
--max-runtime-instances 64
--max-parallel-downloads 7
--prometheus-port 19635
--state-cache-size 0
--port 30333
--ws-port 9934
--rpc-port 9933
--
--execution wasm
--port 30334
--ws-port 9936
--rpc-port 9935