Node stuck on block 4,470,069
Closed this issue · 19 comments
ligi commented
after the recent validation my node is stuck (was working for > 30 epochs before)
looks like this for ~over 1h now:
I see a lot of `tx can't be accepted due to validation ceremony``in the logs - some more log here:
NFO [05-28|15:01:41.202] rate metric
peer bytesSent duration rate(kb/s) bytesReceived duration rate(kb/s)
QmaQewWn5guHWsPuGQGankRiQNEUvmEkXk8ykrk2uTRh3p 7200 11.844549ms 593.627 54491 29.023186445s 1.833
QmVaQq5u4bTs5ynJS3JTh2FzpsEBQ4fwxfuMaR6dcmDaZj 7552 12.065774ms 611.233 52464 21.03716216s 2.435
QmfUPkeNfh7YS6oQB5PeFAKjaZDdSgEu8uPMUu34ENzoDk 343812 2.499607ms 134322.678 17 46.364654711s 0.000
QmdJTbGFNxXtXxc9xi7mCXVnGfjZTdjzDbDxvXFsPCDRcs 0 0s 0.000 62 246.230507ms 0.246
QmVK5L9a1fFPCP4dkKrEY6K99jcSiToNkhcJjrXwGD6NHp 182 350.458µs 507.149 602 5.478085263s 0.107
WARN [05-28|15:01:49.317] Flip Tx is not valid hash=0xf3455d0afa5259fc2e5be426bdf794881a72a2a3b26cc19e6fbd0bbda98be815 err="tx can't be accepted due to validation ceremony"
ERROR[05-28|15:01:49.317] invalid flip err="tx can't be accepted due to validation ceremony"
INFO [05-28|15:01:51.930] peer has been banned id=QmaQewWn5guHWsPuGQGankRiQNEUvmEkXk8ykrk2uTRh3p reason="empty blocks' hashes mismatch"
WARN [05-28|15:01:51.930] Block 4470070 is invalid: empty blocks' hashes mismatch component=downloader
INFO [05-28|15:01:51.930] Peer disconnected id=QmaQewWn5guHWsPuGQGankRiQNEUvmEkXk8ykrk2uTRh3p shardId=0
WARN [05-28|15:01:52.226] Flip Tx is not valid hash=0x7f900c5bf2d1c1bf5983abc3df01b586b819e4b286b85a18bdf2b402782af299 err="tx can't be accepted due to validation ceremony"
ERROR[05-28|15:01:52.226] invalid flip err="tx can't be accepted due to validation ceremony"
INFO [05-28|15:01:52.931] Start process batch component=downloader from=4470070 to=4470168
INFO [05-28|15:01:54.463] Peer connected id=Qmf9unHRRaieRdS9hAp8RpNCYMEkEAXn4KkyQx2CupyBv8 inbound=false shardId=3
INFO [05-28|15:01:55.465] Peer aborts connection id=Qmf9unHRRaieRdS9hAp8RpNCYMEkEAXn4KkyQx2CupyBv8 shardId=3 reason="no slots for shard"
INFO [05-28|15:02:00.017] Peer connected id=QmaqY3MAEe8wdFSiZRNpbTvfjn6Y9n2A8JURiSxyRTrtiV inbound=false shardId=0
INFO [05-28|15:02:00.870] Peer aborts connection id=QmaqY3MAEe8wdFSiZRNpbTvfjn6Y9n2A8JURiSxyRTrtiV shardId=0 reason="no slots for shard"
INFO [05-28|15:02:09.490] Peer connected id=QmUQNJ5RhgyUzX2Hp13FRCFXwof2g19PUP28s52ht6gZXc inbound=false shardId=2
INFO [05-28|15:02:11.203] rate metric
peer bytesSent duration rate(kb/s) bytesReceived duration rate(kb/s)
QmWz19CLxqem2TENXRkhUUK4G6Nfuhga2HXMBMg3BkmHsx 25 74.276µs 328.694 180 1m7.496677475s 0.003
QmVPdratsY9zdP91WcRqnxdKvR6BU5NLZ3Uwuh2yQFcNJW 180 73.236µs 2400.203 25 1m7.579068211s 0.000
Qmf9unHRRaieRdS9hAp8RpNCYMEkEAXn4KkyQx2CupyBv8 82 91.434µs 875.802 89 15.554483ms 5.588
QmaqY3MAEe8wdFSiZRNpbTvfjn6Y9n2A8JURiSxyRTrtiV 82 41.958µs 1908.531 87 5.549006138s 0.015
QmTk4bFzTs2yZ7vyRkcSqX6Ha1XGG21Co7mCJpgt4U8g4x 180 84.234µs 2086.821 25 1m7.482331475s 0.000
QmVK5L9a1fFPCP4dkKrEY6K99jcSiToNkhcJjrXwGD6NHp 14872 25.304617ms 573.944 1079630 26.060174735s 40.457
QmVaQq5u4bTs5ynJS3JTh2FzpsEBQ4fwxfuMaR6dcmDaZj 14830 24.299957ms 595.985 168913 32.457011645s 5.082
Qmarajhga16EtfqiQMwTAVUCHUtk9ipiUJzUaE2DnfQYjt 29 46.797µs 605.174 29 1m7.47267047s 0.000
QmWf7SzjqiXBtVfqWYcmTLW79nTWGxdVpaipSoMo43BNHG 180 162.03µs 1084.869 25 38.879100682s 0.001
QmaQewWn5guHWsPuGQGankRiQNEUvmEkXk8ykrk2uTRh3p 7605 11.709197ms 634.267 53198 15.995644599s 3.248
QmdtC9S3LRF4Rb66NcrCx2jALAj6mjJamM3qM6UEPiUx7P 25 76.555µs 318.909 180 1m7.446240333s 0.003
QmUPPTCXq4NM8R31LYJpwCqyFfb7FNkZXnVFZCWmQEU3mT 180 104.874µs 1676.118 25 1m7.730292994s 0.000
QmfUPkeNfh7YS6oQB5PeFAKjaZDdSgEu8uPMUu34ENzoDk 205 170.709µs 1172.729 27806 32.738708094s 0.829
QmUQNJ5RhgyUzX2Hp13FRCFXwof2g19PUP28s52ht6gZXc 82 60.837µs 1316.273 146 44.640656ms 3.194
INFO [05-28|15:02:11.933] Total validation reward reward=545232
INFO [05-28|15:02:19.734] Shard distribution before balancing shardId=1 newbies=811 verified,human=2377 suspended,zombie=832
INFO [05-28|15:02:19.735] Shard distribution before balancing shardId=2 newbies=803 verified,human=2386 suspended,zombie=818
INFO [05-28|15:02:19.735] Shard distribution before balancing shardId=3 newbies=807 verified,human=2439 suspended,zombie=792
INFO [05-28|15:02:19.735] Shard distribution before balancing shardId=4 newbies=786 verified,human=2483 suspended,zombie=798
INFO [05-28|15:02:22.689] Shard distribution after balancing shardId=1 newbies=801 verified,human=2422 suspended,zombie=810
INFO [05-28|15:02:22.690] Shard distribution after balancing shardId=2 newbies=802 verified,human=2421 suspended,zombie=810
INFO [05-28|15:02:22.690] Shard distribution after balancing shardId=3 newbies=802 verified,human=2421 suspended,zombie=810
INFO [05-28|15:02:22.690] Shard distribution after balancing shardId=4 newbies=802 verified,human=2421 suspended,zombie=810
INFO [05-28|15:02:22.690] Sharding info prev shards count=4 new shards count=4
WARN [05-28|15:02:24.032] Flip Tx is not valid hash=0x78cc704a8ec1f5f5ef33359c076a43f29c51266d7bc8c39582ddd7dfd62e393f err="tx can't be accepted due to validation ceremony"
ERROR[05-28|15:02:24.033] invalid flip err="tx can't be accepted due to validation ceremony"
rioda-org commented
After node has worked for many epochs and many hard forks, it is good idea to clear ipfs and idenachain.db folders to let it sync from scratch. Refresh it, sort of.
hlolve commented
Many nodes failed
On nodes that were OK after round=4470070 you have in log
INFO [05-28|14:17:36.098] Start loop round=4470069 head=0x486938a1eb2d5555588d4fe8449882df0931f65e887382d22b44beffcf11b5e1
INFO [05-28|14:17:56.079] Start loop round=4470070 head=0xabc8bf3f2fa3216f72980be67221d9bf024ba241bcd06d3a45a05dbede23dafa
INFO [05-28|14:20:53.705] Reached consensus on empty block
INFO [05-28|14:20:53.708] Start loop round=4470071 head=0x04b223019e329b4bf77a7f90ee19af8b8ef6e8b69b5eb54bb4315d434c33bda8On nodes that failed in round=4470070 you have in log
INFO [05-28|14:17:36.096] Start loop round=4470069 head=0x486938a1eb2d5555588d4fe8449882df0931f65e887382d22b44beffcf11b5e1
INFO [05-28|14:17:56.096] Start loop round=4470070 head=0xabc8bf3f2fa3216f72980be67221d9bf024ba241bcd06d3a45a05dbede23dafa
INFO [05-28|14:20:47.895] Block was received successfully hash=0x04b223019e329b4bf77a7f90ee19af8b8ef6e8b69b5eb54bb4315d434c33bda8
ERROR[05-28|14:21:04.915] Add block err="empty blocks' hashes mismatch"
INFO [05-28|14:21:23.061] peer has been banned id=QmREVxN7sNhzA417K8tPmWQ9mgP8dib2dm5BozPVAWfduX reason="empty blocks' hashes mismatch"
WARN [05-28|14:21:23.062] Block 4470070 is invalid: empty blocks' hashes mismatch component=downloader
WARN [05-28|14:21:24.063] failed to process batch component=downloader err="Batch (4470070-4470070) can't be loaded"
hlolve commented
hlolve commented
hlolve commented
Same error today on some nodes
ERROR[06-18|14:16:37.347] Add block err="empty blocks' hashes mismatch"
WARN [06-18|14:16:55.944] Block 4560912 is invalid: empty blocks' hashes mismatch component=downloader
EggPool commented
Same error for me last validation session.
Had to clear ipfs and idenachain.db, resync'd ok.
Stuck again this validation.
I was on 29.0.
Cleared ipfs and idenachain.db, resync'd up to 4 560 911 and stuck again.
Upgraded to 29.1, still stuck.
INFO [06-18|16:06:05.676] peer has been banned id=QmUQwp8HH7yr2MqZeHNxoif5X3Sxw9sFGjsVzkoz1JuyC4 reason="empty blocks' hashes mismatch"
WARN [06-18|16:06:05.676] Block 4560912 is invalid: empty blocks' hashes mismatch component=downloader
Banning peer after peer for the same reason.
midenaio commented
The issue has been reproduced. The root cause of the problem is in the differences in the floating point operations on some computers. This leads to the fact that some nodes calculate a bit different amounts of the staking rewards (stake^0.9) and generate blocks with different hashes. These nodes get stuck because they cannot accept the block provided by the rest of the network. The issue can be fixed only with a hard fork update.
Unfortunately on the next validation, the nodes will again be stuck on the last block of the epoch. Once it happens you will need to wait until the remaining network generates the state snapshot (snapshot is generated every 1000 blocks) so the fast sync can be used to synchronize your node.
ubiubi18 commented
As i write this idena.io/staking does calculate for a stake of 1010 idna a reward of 22.99 idna, but for a stake of 1011 idna a reward of just 22.1 idna... for 51 staked idna it calculates reward of 1.50 idna, but for 52 staked idna suddenly 2.52 idna?!? Is this related to floating points issue in calculations as well, or is it a different bug/issue?
ligi commented
I can confirm - it happened again in epoch 90 - now stuck at 4,742,620
ubiubi18 commented
same issue, though my client with inbuild node keeps verifying blocks and has no mining penalty yet, no matter that it is out of sync for multiple hours now. IT said for a short time: "Synchronized", but was stuck at "Current block: 4742620" and also the message "Please wait. The network is reaching consensus on validated identities" - But at the same time i am verifying blocks according the the block explorers balance history of my address.
pocoloko commented
has no mining penalty yet
you will get the penalty as soon as your node actually becomes online.
So I've had this happen again this validation, but this time I cannot recover. I deleted ipfs and idenachain.db folder and let the node sync again, and it got stuck again, this is the last of the syncing:
INFO [07-30|17:29:14.828] Start loading of snapshot component=downloader height=4742465
INFO [07-30|17:29:14.932] Snapshot loading progress=15.000001%
INFO [07-30|17:29:15.430] Snapshot loading progress=30.000002%
INFO [07-30|17:29:16.435] Snapshot loading progress=50%
INFO [07-30|17:29:17.364] Snapshot loading progress=75%
INFO [07-30|17:29:17.985] Snapshot has been loaded component=downloader height=4742465
INFO [07-30|17:29:20.083] Peer aborts connection id=QmRKXg4ccYTpwiQQtw3bXAxW7wsj5tH7oUkQ9esxYfM1SL shardId=4 reason="peer was selected to disconnect while rene
INFO [07-30|17:29:23.134] Peer connected id=QmdaHr1sh1VFX7RNHpWM2QiL5Ty4LDx4V5tJRpPnFw3afk inbound=true shardId=1
INFO [07-30|17:29:23.371] Found manifest component=downloader height=4742465
INFO [07-30|17:29:23.371] Full sync will be used component=downloader
INFO [07-30|17:29:23.371] All blocks were requested. Wait for applying of blocks component=downloader
INFO [07-30|17:29:23.371] Start process batch component=downloader from=4742466 to=4742666
and then it just keeps repeating the shard distribution and peers connect and disconnect all the time...
INFO [07-30|18:19:31.175] Total validation reward reward=545052
INFO [07-30|18:19:34.490] Shard distribution before balancing shardId=1 newbies=701 verified,human=1983 suspended,zombie=711
INFO [07-30|18:19:34.490] Shard distribution before balancing shardId=2 newbies=700 verified,human=1978 suspended,zombie=707
INFO [07-30|18:19:34.490] Shard distribution before balancing shardId=3 newbies=712 verified,human=2011 suspended,zombie=662
INFO [07-30|18:19:34.490] Shard distribution before balancing shardId=4 newbies=676 verified,human=1967 suspended,zombie=721
INFO [07-30|18:19:35.759] Shard distribution after balancing shardId=1 newbies=697 verified,human=1985 suspended,zombie=701
INFO [07-30|18:19:35.759] Shard distribution after balancing shardId=2 newbies=697 verified,human=1985 suspended,zombie=700
INFO [07-30|18:19:35.759] Shard distribution after balancing shardId=3 newbies=697 verified,human=1985 suspended,zombie=700
INFO [07-30|18:19:35.759] Shard distribution after balancing shardId=4 newbies=698 verified,human=1984 suspended,zombie=700
INFO [07-30|18:19:35.759] Sharding info prev shards count=4 new shards count=4
INFO [07-30|18:19:36.868] Peer connected id=QmY1hwxCdM1qZFbF9qmUAVYvwm5HUwB3TpV9pKKoPghdXb inbound=true shardId=2
INFO [07-30|18:19:38.186] Peer aborts connection id=QmY1hwxCdM1qZFbF9qmUAVYvwm5HUwB3TpV9pKKoPghdXb shardId=2 reason="no slots for shard"
INFO [07-30|18:19:39.958] Peer connected id=QmYJcD1EYw6JzGYDSUdNdAUQCR8CmDYXx7bQn52FgfmgNU inbound=true shardId=1
INFO [07-30|18:19:41.016] Peer aborts connection id=QmYJcD1EYw6JzGYDSUdNdAUQCR8CmDYXx7bQn52FgfmgNU shardId=1 reason="no slots for shard"
INFO [07-30|18:19:42.463] Peer connected id=Qmb4KSVFAk16j2vRzkn6iSxLmxkYoghj2LABsfCDPPpCac inbound=true shardId=1
INFO [07-30|18:19:43.293] peer has been banned id=QmaWJfMvKnuEJrJmtjLM78HXsKMjqbQPVvaAR5iNyeAnoi reason="empty blocks' hashes mismatch"
WARN [07-30|18:19:43.293] Block 4742621 is invalid: empty blocks' hashes mismatch component=downloader
INFO [07-30|18:19:43.293] Peer disconnected id=QmaWJfMvKnuEJrJmtjLM78HXsKMjqbQPVvaAR5iNyeAnoi shardId=1
INFO [07-30|18:19:43.495] Peer connected id=QmSLcu5rtQtA5ZKcMbr9T65gw1aWegjscdtaNi6uu9RHdo inbound=false shardId=3
INFO [07-30|18:19:43.538] Peer aborts connection id=Qmb4KSVFAk16j2vRzkn6iSxLmxkYoghj2LABsfCDPPpCac shardId=1 reason="no slots for shard"
INFO [07-30|18:19:44.294] Start process batch component=downloader from=4742621 to=4742821
INFO [07-30|18:19:51.035] Peer connected id=QmcrNhVQ1TmQQsHjJ5r5j9qiKiqiYR1foeDM8HfiKTTa6X inbound=true shardId=1
INFO [07-30|18:19:52.071] Peer aborts connection id=QmcrNhVQ1TmQQsHjJ5r5j9qiKiqiYR1foeDM8HfiKTTa6X shardId=1 reason="no slots for shard"
INFO [07-30|18:19:53.572] Total validation reward reward=545052
INFO [07-30|18:19:56.963] Shard distribution before balancing shardId=1 newbies=701 verified,human=1983 suspended,zombie=711
INFO [07-30|18:19:56.963] Shard distribution before balancing shardId=2 newbies=700 verified,human=1978 suspended,zombie=707
INFO [07-30|18:19:56.963] Shard distribution before balancing shardId=3 newbies=712 verified,human=2011 suspended,zombie=662
INFO [07-30|18:19:56.963] Shard distribution before balancing shardId=4 newbies=676 verified,human=1967 suspended,zombie=721
INFO [07-30|18:19:58.263] Shard distribution after balancing shardId=1 newbies=697 verified,human=1985 suspended,zombie=701
INFO [07-30|18:19:58.264] Shard distribution after balancing shardId=2 newbies=697 verified,human=1985 suspended,zombie=700
INFO [07-30|18:19:58.264] Shard distribution after balancing shardId=3 newbies=697 verified,human=1985 suspended,zombie=700
INFO [07-30|18:19:58.264] Shard distribution after balancing shardId=4 newbies=698 verified,human=1984 suspended,zombie=700
INFO [07-30|18:19:58.265] Sharding info prev shards count=4 new shards count=4
INFO [07-30|18:20:00.401] Peer connected id=QmPrjrdxUiX965LKex6jmZ1azePtndfv7FFRnEaAph5Lbg inbound=true shardId=3
INFO [07-30|18:20:00.602] Peer connected id=QmVyqQLBEVFP6xDAn2NsuSbEAKKLhzUZgmJ45H7sAawdWq inbound=true shardId=0
INFO [07-30|18:20:00.987] Peer connected id=QmQE5LKRcdcK7xqmseU7W7oC6jmy72VDp7oqTWENikALyg inbound=true shardId=1
INFO [07-30|18:20:01.470] Peer aborts connection id=QmPrjrdxUiX965LKex6jmZ1azePtndfv7FFRnEaAph5Lbg shardId=3 reason="no slots for shard"
INFO [07-30|18:20:01.708] Peer aborts connection id=QmVyqQLBEVFP6xDAn2NsuSbEAKKLhzUZgmJ45H7sAawdWq shardId=0 reason="no slots for shard"
Why is this taking 2 months to fix?
ligi commented
So I've had this happen again this validation, but this time I cannot recover. I deleted ipfs and idenachain.db folder and let the node sync again, and it got stuck again, this is the last of the syncing:
I think you have to wait 1000 blocks before resyncing. So resync when block 4,743,620 is reached (this is what I will try at least)
pocoloko commented
Syncing after 1000 blocks worked, node running again, this still needs to be fixed though
ligi commented
this epoch start I did not need to resync - which surprised me actually as the fix was merged 2 days ago and hence is not yet on my node. But maybe we can still close this one then?!
midenaio commented
@ligi did your node run on the same machine as usual? If so could you share your node log file or send it to info@idena.io?
The fix hasn't been released yet. It will be included in the upcoming hard fork update.
pocoloko commented
something was different this time... my shared nodes didn't get stuck, or well they did but they unstuck themselves.
This is the reaching of consensus and start of stuck node:
INFO [08-20|14:15:57.094] Approved candidates shardId=1 cnt=2660
INFO [08-20|14:15:58.414] Approved candidates shardId=2 cnt=2709
INFO [08-20|14:15:59.547] Approved candidates shardId=3 cnt=2702
INFO [08-20|14:16:00.663] Approved candidates shardId=4 cnt=2639
INFO [08-20|14:16:05.268] Total validation reward reward=544368
INFO [08-20|14:16:09.274] Shard distribution before balancing shardId=1 newbies=676 verified,human=1865 suspended,zombie=623
INFO [08-20|14:16:09.274] Shard distribution before balancing shardId=2 newbies=676 verified,human=1881 suspended,zombie=626
INFO [08-20|14:16:09.274] Shard distribution before balancing shardId=3 newbies=694 verified,human=1862 suspended,zombie=625
INFO [08-20|14:16:09.274] Shard distribution before balancing shardId=4 newbies=635 verified,human=1839 suspended,zombie=634
INFO [08-20|14:16:10.739] Shard distribution after balancing shardId=1 newbies=670 verified,human=1862 suspended,zombie=627
INFO [08-20|14:16:10.739] Shard distribution after balancing shardId=2 newbies=670 verified,human=1862 suspended,zombie=627
INFO [08-20|14:16:10.740] Shard distribution after balancing shardId=3 newbies=670 verified,human=1862 suspended,zombie=627
INFO [08-20|14:16:10.740] Shard distribution after balancing shardId=4 newbies=671 verified,human=1861 suspended,zombie=627
INFO [08-20|14:16:10.740] Sharding info prev shards count=4 new shards count=4
INFO [08-20|14:16:20.281] Wait for block proposal
INFO [08-20|14:16:27.419] Total validation reward reward=544368
INFO [08-20|14:16:31.506] Shard distribution before balancing shardId=1 newbies=676 verified,human=1865 suspended,zombie=623
INFO [08-20|14:16:31.506] Shard distribution before balancing shardId=1 newbies=676 verified,human=1865 suspended,zombie=623
INFO [08-20|14:16:31.506] Shard distribution before balancing shardId=2 newbies=676 verified,human=1881 suspended,zombie=626
INFO [08-20|14:16:31.506] Shard distribution before balancing shardId=3 newbies=694 verified,human=1862 suspended,zombie=625
INFO [08-20|14:16:31.506] Shard distribution before balancing shardId=4 newbies=635 verified,human=1839 suspended,zombie=634
INFO [08-20|14:16:33.053] Shard distribution after balancing shardId=1 newbies=670 verified,human=1862 suspended,zombie=627
INFO [08-20|14:16:33.053] Shard distribution after balancing shardId=2 newbies=670 verified,human=1862 suspended,zombie=627
INFO [08-20|14:16:33.053] Shard distribution after balancing shardId=3 newbies=670 verified,human=1862 suspended,zombie=627
INFO [08-20|14:16:33.053] Shard distribution after balancing shardId=4 newbies=671 verified,human=1861 suspended,zombie=627
INFO [08-20|14:16:33.053] Sharding info prev shards count=4 new shards count=4
ERROR[08-20|14:16:42.299] Proposed block is not found err="invalid block roots. Expected=66311dc3ea8782096bfc20d85b803c3097b868e022f02d49022099fff6694fad & bc2274dd1091774f632d54891b5e088170dc83389eee365dd2819ec8bc6e1fd8, actual=4ae8a730bf35fd72eb0e3c346b158cee2fa914770609a6e28541f24b8eacb73c & bc2274dd1091774f632d54891b5e088170dc83389eee365dd2819ec8bc6e1fd8"
then it went through an endless loop of peer connected/disconnected/aborts connection and also repeating the Shard distribution messages until...
INFO [08-20|15:19:33.876] Shard distribution before balancing shardId=1 newbies=676 verified,human=1865 suspended,zombie=623
INFO [08-20|15:19:33.876] Shard distribution before balancing shardId=2 newbies=676 verified,human=1881 suspended,zombie=626
INFO [08-20|15:19:33.877] Shard distribution before balancing shardId=3 newbies=694 verified,human=1862 suspended,zombie=625
INFO [08-20|15:19:33.877] Shard distribution before balancing shardId=4 newbies=635 verified,human=1839 suspended,zombie=634
INFO [08-20|15:19:35.338] Shard distribution after balancing shardId=1 newbies=670 verified,human=1862 suspended,zombie=627
INFO [08-20|15:19:35.338] Shard distribution after balancing shardId=2 newbies=670 verified,human=1862 suspended,zombie=627
INFO [08-20|15:19:35.339] Shard distribution after balancing shardId=3 newbies=670 verified,human=1862 suspended,zombie=627
INFO [08-20|15:19:35.339] Shard distribution after balancing shardId=4 newbies=671 verified,human=1861 suspended,zombie=627
INFO [08-20|15:19:35.339] Sharding info prev shards count=4 new shards count=4
INFO [08-20|15:19:45.103] peer has been banned id=QmXGfKWZyfPftecrWHoksS5ttT6phYip8ZzPrazWs3PE38 reason="empty blocks' hashes mismatch"
WARN [08-20|15:19:45.103] Block 4833349 is invalid: empty blocks' hashes mismatch component=downloader
INFO [08-20|15:19:45.104] Peer disconnected id=QmXGfKWZyfPftecrWHoksS5ttT6phYip8ZzPrazWs3PE38 shardId=1
INFO [08-20|15:19:46.104] Start process batch component=downloader from=4833349 to=4833456
WARN [08-20|15:19:46.105] failed to process batch component=downloader err="number of attempts exceeded limit"
INFO [08-20|15:19:46.105] Found manifest component=downloader height=4833465
INFO [08-20|15:19:46.105] Fast sync will be used component=downloader
INFO [08-20|15:19:46.491] All blocks were requested. Wait for applying of blocks component=downloader
INFO [08-20|15:19:46.491] Start process batch component=downloader from=4833349 to=4833465
INFO [08-20|15:19:47.178] Peer aborts connection id=QmRFmrxrkSN4xjQzoNJqyNTUF96TktcLraboybS42ZwyWD shardId=1 reason="unnecessary shard"
INFO [08-20|15:19:47.276] Peer connected id=QmNzJYCS3Dz7r6N6rEVqT1WzGRjq4ZQSyomJzGVCY7wN8u inbound=true shardId=3
INFO [08-20|15:19:49.628] Finish process batch component=downloader from=4833349 to=4833465
INFO [08-20|15:19:49.628] Start process batch component=downloader from=4833466 to=4833465
INFO [08-20|15:19:49.628] Finish process batch component=downloader from=4833466 to=4833465
INFO [08-20|15:19:49.628] Start process batch component=downloader from=4833466 to=4833465
INFO [08-20|15:19:49.628] Finish process batch component=downloader from=4833466 to=4833465
INFO [08-20|15:19:49.628] Start loading of snapshot component=downloader height=4833465
INFO [08-20|15:19:50.621] Snapshot loading progress=15.000001%
INFO [08-20|15:19:50.623] Snapshot loading progress=30.000002%
INFO [08-20|15:19:53.403] Snapshot loading progress=50%
<snip>
INFO [08-20|15:19:57.800] Snapshot loading progress=75%
INFO [08-20|15:19:58.155] Snapshot has been loaded component=downloader height=4833465
<snip>
INFO [08-20|15:20:03.650] Short session timer has been created time=2022-09-10T13:30:00+0000
INFO [08-20|15:20:03.650] Full sync will be used component=downloader
INFO [08-20|15:20:03.650] All blocks were requested. Wait for applying of blocks component=downloader
INFO [08-20|15:20:03.651] Start process batch component=downloader from=4833466 to=4833470
INFO [08-20|15:20:04.169] Peer connected id=QmQtpTyQx7d5jHF7mAxJC8nhASyK9sSS6wkYTLeErAeqcd inbound=true shardId=2
INFO [08-20|15:20:05.752] Peer connected id=QmVBZmSzPgws9P6ffSr1NaVZijbf4DPTgmjFcjA848p3cJ inbound=true shardId=2
INFO [08-20|15:20:05.814] Finish process batch component=downloader from=4833466 to=4833470
INFO [08-20|15:20:05.814] Start process batch component=downloader from=4833471 to=4833471
INFO [08-20|15:20:06.045] Peer aborts connection id=QmVBZmSzPgws9P6ffSr1NaVZijbf4DPTgmjFcjA848p3cJ shardId=2 reason="no slots for shard"
INFO [08-20|15:20:06.712] Peer connected id=QmZkSoP7D3UGZPYytzArmj8ap1Y1LMiAA1GJdaXfUzAjrw inbound=true shardId=2
INFO [08-20|15:20:07.103] Finish process batch component=downloader from=4833471 to=4833471
INFO [08-20|15:20:07.103] Node is synchronized component=downloader
INFO [08-20|15:20:07.282] Start loop round=4833472 head=0x9c041a244607344f17ed039e59bfb20176e5056da4aa8baa6099c5ad38928a95 shardId=1 p2p-shardId=0
INFO [08-20|15:20:07.770] Peer aborts connection id=QmZkSoP7D3UGZPYytzArmj8ap1Y1LMiAA1GJdaXfUzAjrw shardId=2 reason="no slots for shard"
INFO [08-20|15:20:08.934] Peer connected id=QmQegU54hn9WPyRjhvYYN87ZefqkkXAmXnNxjdGuiYxHfW inbound=true shardId=0
INFO [08-20|15:20:16.194] Peer aborts connection id=QmZeki4dW6ZzipNAY8zi83Efydm4cV39KwVBVRGprG4P1u shardId=4 reason="unnecessary shard"
INFO [08-20|15:20:17.282] Selected proposer proposer="NOT FOUND"
INFO [08-20|15:20:17.296] Reduction started block=0x63d79cd77ebb99ea070d1b6c3fb1fe8f35619b761943694c5164c52a4a6d605b
INFO [08-20|15:20:17.312] Reduction completed block=0x7e3dd8811e1fafe88a2a123e9bb24d9b1cc952f5524f524faa370064c03c132c isEmpty=false
INFO [08-20|15:20:17.312] binaryBa started block=0x7e3dd8811e1fafe88a2a123e9bb24d9b1cc952f5524f524faa370064c03c132c
INFO [08-20|15:20:17.425] Block was received successfully hash=0x7e3dd8811e1fafe88a2a123e9bb24d9b1cc952f5524f524faa370064c03c132c
INFO [08-20|15:20:17.833] Reached FINAL block=0x7e3dd8811e1fafe88a2a123e9bb24d9b1cc952f5524f524faa370064c03c132c txs=4
and thats it, after that it started working as normal...
hlolve commented
I think it will help if with reports is mentioned cpu type used.
I remember from past validations that:
Amd Ryzen not failed
Intel Xeon failed
ARM failed