eth_getTransactionReceipt returns null for all transactions in some blocks
zimbabao opened this issue · 10 comments
Description
Fully synced archive node returns null as a response to eth_getTransactionReceipt for all transactions for some blocks.
This is unexpected, the chain continues to sync fine.
Current Behavior
For this block https://celoscan.io/block/10158797 (and may be more)
I get following response
curl http://localhost:8545 -X POST -H "Content-Type: application/json" --data '{"method":"eth_getTransactionReceipt","params":["0xd426b6bbd686e2588ea8bb20612ba3e78ab37f3b0dc70b955241677832aab4a9"],"id":1,"jsonrpc":"2.0"}'
{"jsonrpc":"2.0","id":1,"result":null}
Config
image: us.gcr.io/celo-org/geth:mainnet
imagePullPolicy: Always
args:
- --syncmode=full
- --datadir=/root/.celo/
- --datadir.ancient=/root/.celo/celo/chaindata/ancient
- --http
- --http.addr=0.0.0.0
- --http.api=eth,net,web3,txpool,admin,debug
- --ws
- --ws.addr=0.0.0.0
- --ws.api=eth,net,web3,txpool,admin,debug
- --metrics
- --metrics.addr=0.0.0.0
- --txlookuplimit=0
- --gcmode=archive
- --light.serve=90
- --light.maxpeers=90
- --maxpeers=300
- --rpc.gascap=300000000 # 300M limit on gas, default is 50M
- --http.vhosts=*
- --cache=200000
- --cache.preimages
- --verbosity=3
Context
Node is synced from genesis. A couple of time disk got full and I increased the disk size and restarted the node.
Hi @zimbabao! Thanks for reporting, I couldn't reproduce it against our archive nodes. Which version are you running?
celo/v1.7.2-stable/linux-amd64/go1.17.13 .
Is there any place from where I can snapshot for archive node?
Hi @zimbabao . I could also not reproduce this. Have you seen this only on the block you mentioned or on other ones as well? My best guess is that something got mangled up in the database and you need to resync.
Is there any place from where I can snapshot for archive node?
I'm not aware of that, but I'll ask around and let you know.
Unfortunately, all my attempts to resync a different node fails after some 2.5 million blocks.
I suspect one section of peer nodes has some problem and I am not able to point it out. Those peers never let the node finish.
Any chance you are aware of any issue?.
A snapshot of the archive node will be very helpful.
Unfortunately, all my attempts to resync a different node fails after some 2.5 million blocks.
I suspect one section of peer nodes has some problem and I am not able to point it out. Those peers never let the node finish.
Any chance you are aware of any issue?.
No, not currently. Does this happen at a certain block? And do you have a way to test or reproduce this behaviour?
A snapshot of the archive node will be very helpful.
Unfortunately we do not have archive snapshots, as the they're just to big to keep.
Thanks, @palango for your reply.
No, it's not on a certain block number.
Currently, I get the issue at block 17805618.
I see errors like this for hours.
Any idea what is the root cause of this issue?
WARN [02-21|15:42:10.129] Synchronisation failed, dropping peer peer=b4fac8f11b0e090ca764097e07752bb40df2ad44e9e9a23b17d59fbf30b7e466 err="retrieved hash chain is invalid: error attempting to retrieve headers: epochSize 17280, headerHash 0xa855e1c4e900c1dff6c33299c10c2889822c0be993160cc365fa4c73fd91783c"
ERROR[02-21|15:42:10.130] Ethereum peer removal failed peer=b4fac8f1 err="peer not registered"
WARN [02-21|15:42:21.283] Synchronisation failed, dropping peer peer=4e36b98c620f75fcfd2d689aafd746104cbee8451d977cfbb6241858902e4574 err=timeout
ERROR[02-21|15:42:21.283] Ethereum peer removal failed peer=4e36b98c err="peer not registered"
WARN [02-21|15:42:33.531] Synchronisation failed, dropping peer peer=d5fd4b19135f05bb29a0ff79a19feb7caf0242e418457a95aac80fc7536e435e err=timeout
ERROR[02-21|15:42:33.531] Ethereum peer removal failed peer=d5fd4b19 err="peer not registered"
WARN [02-21|15:42:44.441] Synchronisation failed, dropping peer peer=31985acbb0b695ee8c37fa6ef0488ecd5a79601fc51722574e00b09b7e6a9e55 err=timeout
ERROR[02-21|15:42:44.441] Ethereum peer removal failed peer=31985acb err="peer not registered"
INFO [02-21|15:42:47.406] Downloader queue stats receiptTasks=0 blockTasks=0 itemSize=1.62KiB throttle=8192
WARN [02-21|15:42:47.426] Synchronisation failed, dropping peer peer=b4fac8f11b0e090ca764097e07752bb40df2ad44e9e9a23b17d59fbf30b7e466 err="retrieved hash chain is invalid: error attempting to retrieve headers: epochSize 17280, headerHash 0xa855e1c4e900c1dff6c33299c10c2889822c0be993160cc365fa4c73fd91783c"
ERROR[02-21|15:42:47.426] Ethereum peer removal failed peer=b4fac8f1 err="peer not registered"
WARN [02-21|15:42:47.536] Synchronisation failed, dropping peer peer=239e8ce634c7bc7cb95776f75f8e3f2213a5ada56083a560dfd3b94a54b7c712 err="action from bad peer ignored: returned headers 192 != requested 1"
ERROR[02-21|15:42:47.536] Ethereum peer removal failed peer=239e8ce6 err="peer not registered"
WARN [02-21|15:43:03.118] Synchronisation failed, dropping peer peer=87523f00c9b096cb7eff88d7d2efca97cedc7ce5c2c841c14bce27ff973575ea err=timeout
ERROR[02-21|15:43:03.118] Ethereum peer removal failed peer=87523f00 err="peer not registered"
WARN [02-21|15:43:14.181] Synchronisation failed, dropping peer peer=14d1f01080ddf2c3801c3ed000add7daf7915e4df34eb6ede48e13a8e67aea09 err=timeout
ERROR[02-21|15:43:14.181] Ethereum peer removal failed peer=14d1f010 err="peer not registered"
To fix null responses. I copies ancient files from a working full node and restarted the archive node.