Ful Sync: BAD BLOCK
elsampsa opened this issue · 4 comments
elsampsa commented
Synopsis
Geth stops working after finding a bad block:
Error: could not apply tx 122 [0x2e91ec8bde129768ad253384dc6002e66cd878de2bc393373998a8c4ac3a456e]: gas limit reached
(please see the whole backlog in the end).
Similar issues:
What I understand from these issues is that, as geth sifts through enormous amounts of data, with more time and data, the probability for a random erroneous bit-flip goes up. And then it happens and then the (whole?) blockchain goes sour.
Do the unclean shutdowns (see the backlog below) increase this probability as well?
So instead of a bug, this is blockchain corruption?
Is there any way to remove the corrupted block .. or is the origin of the problem hidden somewhere deep down in the blockchain and the only way to solve this, is to download the whole thing again? (a think I'd rather avoid since I never got the fast sync to actually sync, see this).
System information
Geth version: 1.10
OS & Version: LInux (Ubuntu 18.04 LTS)
Expected behaviour
Actual behaviour
Geth stops as it finds a "bad block"
Steps to reproduce the behaviour
N/A
Backtrace
INFO [04-16|23:48:11.401] Starting Geth on Ethereum mainnet...
INFO [04-16|23:48:11.405] Maximum peer count ETH=50 LES=0 total=50
INFO [04-16|23:48:11.406] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [04-16|23:48:11.408] Set global gas cap cap=25000000
INFO [04-16|23:48:11.408] Allocated trie memory caches clean=511.00MiB dirty=512.00MiB
INFO [04-16|23:48:11.408] Allocated cache and file handles database=/home/sampsa/.ethereum/geth/chaindata cache=1024.00MiB handles=524288
INFO [04-16|23:48:17.383] Opened ancient database database=/home/sampsa/.ethereum/geth/chaindata/ancient
INFO [04-16|23:48:17.417] Initialised chain configuration config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Berlin: <nil>, YOLO v3: <nil>, Engine: ethash}"
INFO [04-16|23:48:20.487] Disk storage enabled for ethash caches dir=/home/sampsa/.ethereum/geth/ethash count=3
INFO [04-16|23:48:20.487] Disk storage enabled for ethash DAGs dir=/home/sampsa/.ethash count=2
INFO [04-16|23:48:20.498] Initialising Ethereum protocol network=1 dbversion=8
INFO [04-16|23:48:21.073] Loaded most recent local header number=12244002 hash="96f625…3cc4fb" td=23316431817258711456735 age=1d3h39m
INFO [04-16|23:48:21.073] Loaded most recent local full block number=12244002 hash="96f625…3cc4fb" td=23316431817258711456735 age=1d3h39m
INFO [04-16|23:48:21.073] Loaded most recent local fast block number=12244002 hash="96f625…3cc4fb" td=23316431817258711456735 age=1d3h39m
INFO [04-16|23:48:21.090] Loaded local transaction journal transactions=0 dropped=0
INFO [04-16|23:48:21.090] Regenerated local transaction journal transactions=0 accounts=0
WARN [04-16|23:48:21.102] Unclean shutdown detected booted=2021-03-06T11:38:58+0200 age=1mo1w4d
WARN [04-16|23:48:21.102] Unclean shutdown detected booted=2021-03-10T22:27:41+0200 age=1mo1w20m
WARN [04-16|23:48:21.102] Unclean shutdown detected booted=2021-03-18T19:50:26+0200 age=4w1d2h
WARN [04-16|23:48:21.103] Unclean shutdown detected booted=2021-03-19T21:34:14+0200 age=4w1h14m
WARN [04-16|23:48:21.103] Unclean shutdown detected booted=2021-03-24T19:33:27+0200 age=3w2d3h
WARN [04-16|23:48:21.103] Unclean shutdown detected booted=2021-04-02T13:21:11+0300 age=2w10h27m
WARN [04-16|23:48:21.103] Unclean shutdown detected booted=2021-04-03T11:16:43+0300 age=1w6d12h
INFO [04-16|23:48:21.103] Starting peer-to-peer node instance=Geth/v1.10.0-stable-56dec25a/linux-amd64/go1.15.6
INFO [04-16|23:48:21.203] New local node record seq=231 id=42e508921cacda09 ip=127.0.0.1 udp=30303 tcp=30303
INFO [04-16|23:48:21.203] Started P2P networking self=enode://46eaa513d960e6edaa88575ad966c46ad75fea4e39c1eddd6cfe522a6341f05c9358d0421ceb5a55b4670ecbc3b2fb986a9f301c889339f5d1d70b04b28875d7@127.0.0.1:30303
INFO [04-16|23:48:21.205] IPC endpoint opened url=/home/sampsa/.ethereum/geth.ipc
INFO [04-16|23:48:23.662] New local node record seq=232 id=42e508921cacda09 ip=84.248.88.68 udp=30303 tcp=30303
INFO [04-16|23:48:31.865] Looking for peers peercount=4 tried=44 static=0
INFO [04-16|23:48:41.899] Looking for peers peercount=2 tried=48 static=0
INFO [04-16|23:48:52.107] Looking for peers peercount=5 tried=45 static=0
INFO [04-16|23:49:02.157] Looking for peers peercount=20 tried=50 static=0
INFO [04-16|23:49:12.166] Looking for peers peercount=20 tried=50 static=0
INFO [04-16|23:49:22.491] Looking for peers peercount=25 tried=50 static=0
INFO [04-16|23:49:32.491] Looking for peers peercount=24 tried=42 static=0
WARN [04-16|23:50:28.181] Checkpoint challenge timed out, dropping id=c1c7674136926899 conn=inbound addr=195.211.153.101:34594 type=OpenEthereum/v3.1.1-...
ERROR[04-16|23:50:28.181] Ethereum peer removal failed peer=c1c76741 err="peer not registered"
INFO [04-16|23:59:36.508] Block synchronisation started
INFO [04-16|23:59:58.658] Downloader queue stats receiptTasks=0 blockTasks=0 itemSize=5.86KiB throttle=8192
INFO [04-16|23:59:59.024] Sidechain written to disk start=12244000 end=12244001 sidetd=23316426079725045704185 localtd=23316431817258711456735
INFO [04-16|23:59:59.538] Importing sidechain segment start=12243876 end=12244005
INFO [04-17|00:00:08.970] Imported new chain segment blocks=3 txs=431 mgas=37.542 elapsed=9.432s mgasps=3.980 number=12243878 hash="42f4ce…59c635" age=1d11h18m dirty=4.88MiB
INFO [04-17|00:00:18.397] Imported new chain segment blocks=3 txs=425 mgas=37.267 elapsed=9.427s mgasps=3.953 number=12243881 hash="ff9acf…826bfa" age=1d11h18m dirty=10.36MiB
INFO [04-17|00:00:28.956] Imported new chain segment blocks=3 txs=486 mgas=37.551 elapsed=10.559s mgasps=3.556 number=12243884 hash="63491c…74a594" age=1d11h18m dirty=15.66MiB
ERROR[04-17|00:00:51.919] Ethereum peer removal failed peer=425f6a40 err="peer not registered"
INFO [04-17|00:00:54.381] Imported new chain segment blocks=3 txs=663 mgas=37.532 elapsed=25.424s mgasps=1.476 number=12243887 hash="6dd100…203d05" age=1d11h18m dirty=21.36MiB
INFO [04-17|00:01:03.272] Imported new chain segment blocks=5 txs=943 mgas=62.511 elapsed=8.890s mgasps=7.031 number=12243892 hash="abf749…04fa47" age=1d11h16m dirty=30.25MiB
INFO [04-17|00:01:13.146] Imported new chain segment blocks=6 txs=788 mgas=62.010 elapsed=9.873s mgasps=6.280 number=12243898 hash="4b9544…d7e661" age=1d11h16m dirty=39.06MiB
INFO [04-17|00:01:21.828] Imported new chain segment blocks=5 txs=1006 mgas=62.417 elapsed=8.681s mgasps=7.189 number=12243903 hash="ee0e53…c9f232" age=1d11h15m dirty=48.26MiB
INFO [04-17|00:01:30.515] Imported new chain segment blocks=6 txs=896 mgas=74.888 elapsed=8.687s mgasps=8.620 number=12243909 hash="20c84b…b61f3a" age=1d11h14m dirty=57.94MiB
INFO [04-17|00:01:39.002] Imported new chain segment blocks=7 txs=1147 mgas=86.826 elapsed=8.487s mgasps=10.230 number=12243916 hash="924c26…f41bab" age=1d11h13m dirty=69.67MiB
INFO [04-17|00:01:50.733] Imported new chain segment blocks=6 txs=1015 mgas=74.916 elapsed=11.730s mgasps=6.386 number=12243922 hash="ec004d…24144d" age=1d11h11m dirty=79.03MiB
INFO [04-17|00:01:59.928] Imported new chain segment blocks=7 txs=1031 mgas=87.493 elapsed=9.194s mgasps=9.515 number=12243929 hash="8cdb7f…b85302" age=1d11h10m dirty=90.27MiB
INFO [04-17|00:02:08.658] Imported new chain segment blocks=6 txs=1054 mgas=74.879 elapsed=8.730s mgasps=8.577 number=12243935 hash="583fad…11b657" age=1d11h9m dirty=101.01MiB
INFO [04-17|00:02:17.643] Imported new chain segment blocks=4 txs=573 mgas=49.884 elapsed=8.984s mgasps=5.552 number=12243939 hash="d03e0f…e881a3" age=1d11h8m dirty=107.69MiB
INFO [04-17|00:02:26.580] Imported new chain segment blocks=8 txs=1089 mgas=99.309 elapsed=8.936s mgasps=11.112 number=12243947 hash="144eed…fdc690" age=1d11h7m dirty=120.88MiB
INFO [04-17|00:02:35.469] Imported new chain segment blocks=8 txs=928 mgas=99.564 elapsed=8.889s mgasps=11.200 number=12243955 hash="57d564…81c335" age=1d11h5m dirty=133.39MiB
INFO [04-17|00:02:44.549] Imported new chain segment blocks=7 txs=1380 mgas=87.358 elapsed=9.080s mgasps=9.621 number=12243962 hash="60542f…57e059" age=1d11h3m dirty=145.74MiB
INFO [04-17|00:02:53.111] Imported new chain segment blocks=10 txs=1558 mgas=124.122 elapsed=8.561s mgasps=14.497 number=12243972 hash="389fea…96c61f" age=1d11h2m dirty=162.32MiB
INFO [04-17|00:03:01.999] Imported new chain segment blocks=6 txs=1134 mgas=74.896 elapsed=8.888s mgasps=8.427 number=12243978 hash="79bff2…1401e4" age=1d11h23s dirty=173.03MiB
INFO [04-17|00:03:10.331] Imported new chain segment blocks=10 txs=1158 mgas=124.494 elapsed=8.331s mgasps=14.942 number=12243988 hash="2f6744…2cbf08" age=1d10h59m dirty=188.37MiB
INFO [04-17|00:03:18.467] Imported new chain segment blocks=7 txs=1173 mgas=87.290 elapsed=8.136s mgasps=10.729 number=12243995 hash="679f3d…df8daa" age=1d10h57m dirty=199.74MiB
INFO [04-17|00:03:23.080] Skip duplicated bad block number=12244000 hash="163838…c42b21"
ERROR[04-17|00:03:23.080]
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Berlin: <nil>, YOLO v3: <nil>, Engine: ethash}
Number: 12244000
Hash: 0x1638380ab737e0e916bd1c7f23bd2bab2a532e44b90047f045f262ee21c42b21
Error: could not apply tx 122 [0x2e91ec8bde129768ad253384dc6002e66cd878de2bc393373998a8c4ac3a456e]: gas limit reached
##############################
WARN [04-17|00:03:23.080] Synchronisation failed, dropping peer peer=99f531968015743ac8aff0f2873156be9d73251f4d88b50e652552e59b82db2f err="retrieved hash chain is invalid: no peers available or all tried for download"
ERROR[04-17|00:03:23.080] Ethereum peer removal failed peer=99f53196 err="peer not registered"
INFO [04-17|00:03:35.875] Downloader queue stats receiptTasks=0 blockTasks=3304 itemSize=14.00KiB throttle=4683
INFO [04-17|00:03:35.939] Importing sidechain segment start=12244000 end=12244009
INFO [04-17|00:03:36.004] Skip duplicated bad block number=12244000 hash="163838…c42b21"
ERROR[04-17|00:03:36.004]
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Berlin: <nil>, YOLO v3: <nil>, Engine: ethash}
Number: 12244000
Hash: 0x1638380ab737e0e916bd1c7f23bd2bab2a532e44b90047f045f262ee21c42b21
Error: could not apply tx 122 [0x2e91ec8bde129768ad253384dc6002e66cd878de2bc393373998a8c4ac3a456e]: gas limit reached
##############################
WARN [04-17|00:03:36.004] Synchronisation failed, dropping peer peer=0ec3556e4394dd8e6a722bb93ab11751842a9c34c16b4043c44007b37831ff20 err="retrieved hash chain is invalid: could not apply tx 122 [0x2e91ec8bde129768ad253384dc6002e66cd878de2bc393373998a8c4ac3a456e]: gas limit reached"
ERROR[04-17|00:03:36.004] Ethereum peer removal failed peer=0ec3556e err="peer not registered"
WARN [04-17|00:05:48.574] Synchronisation failed, dropping peer peer=844e984310fb16ecbcfe7f2ced0284dbd25014d304cf828ad2b8a0956cd4ddb9 err=timeout
ERROR[04-17|00:05:48.574] Ethereum peer removal failed peer=844e9843 err="peer not registered"
Texnomic commented
I'm facing the same issue on the same exact block
ERROR[04-17|09:36:21.648]
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Berlin: <nil>, YOLO v3: <nil>, Engine: ethash}
Number: 12244000
Hash: 0x1638380ab737e0e916bd1c7f23bd2bab2a532e44b90047f045f262ee21c42b21
Error: could not apply tx 122 [0x2e91ec8bde129768ad253384dc6002e66cd878de2bc393373998a8c4ac3a456e]: gas limit reached
############################## ligi commented
This block is the Berlin HardFork - please read: https://blog.ethereum.org/2021/03/08/ethereum-berlin-upgrade-announcement/ and install the latest geth
Texnomic commented
This block is the Berlin HardFork - please read: https://blog.ethereum.org/2021/03/08/ethereum-berlin-upgrade-announcement/ and install the latest geth
I confirm this fixed the issue for me.
elsampsa commented
Fixed for me as well