okx/exchain

Cannot sync node with v1.6.5.1

Closed this issue · 4 comments

dandavid3000 commented

Summary of Bug

1. Describe

exchain client stopped because of the new release and hard fork.
After upgrading exchaind to v1.6.5.1, it's impossible to run the node
(the same error when the old client version stopped).

2. Impact

Impossible to run the node.

Expected Result

The node should be running normally after the new version

Version

v1.6.5.1

Steps to Reproduce

exchaind start --rest.laddr "tcp://localhost:8345" --wsport 8346 --db_backend rocksdb --chain-id exchain-66 --home ${EXCHAIND_PATH}

. module=main 
I[2022-11-11|13:03:29.227][580372] startInProcess. module=main ConsensusStateChainID=exchain-66 GenesisDocChainID=exchain-66
E[2022-11-11|13:03:29.766][580372] dialing failed (attempts: 1): auth failure: secret conn failed: read tcp 192.168.1.135:50432->221.219.103.250:2761: read: connection reset by peer. module=pex addr=b120a8ab901562e62a33e45a2d0c92de948370a0@221.219.103.250:2761
E[2022-11-11|13:03:29.941][580372] Stopping peer for error. module=p2p peer="Peer{MConn{3.16.103.80:26656} e926c8154a2af4390de02303f0977802f15eafe2 out}" err=EOF
E[2022-11-11|13:03:30.166][580372] Stopping peer for error. module=p2p peer="Peer{MConn{175.41.191.69:26656} 7fa5b1d1f1e48659fa750b6aec702418a0e75f13 out}" err=EOF
E[2022-11-11|13:03:30.287][580372] dialing failed (attempts: 1): dial tcp 3.112.119.135:26656: i/o timeout. module=pex addr=c88044fb164896bd9ed29bbee7c290c6ac362133@3.112.119.135:26656
E[2022-11-11|13:03:30.287][580372] dialing failed (attempts: 1): dial tcp 3.131.251.152:26656: i/o timeout. module=pex addr=765249f3e8c40fd918b0662c67b721344cd409e6@3.131.251.152:26656
E[2022-11-11|13:03:30.325][580372] Stopping peer for error. module=p2p peer="Peer{MConn{35.74.8.189:26656} c8f32b793871b56a11d94336d9ce6472f893524b out}" err=EOF
E[2022-11-11|13:03:30.826][580372] Stopping peer for error. module=p2p peer="Peer{MConn{3.16.103.80:26656} e926c8154a2af4390de02303f0977802f15eafe2 out}" err=EOF
E[2022-11-11|13:03:30.942][580372] dialing failed (attempts: 1): dial tcp 3.14.230.237:26656: i/o timeout. module=pex addr=7ce8521efb33ad18523d0d4fb14c2bcb5b1dcf55@3.14.230.237:26656
E[2022-11-11|13:03:30.942][580372] dialing failed (attempts: 1): dial tcp 54.248.38.208:26656: i/o timeout. module=pex addr=7e84f8174c4d693ad53b7ffe679d11d52034f994@54.248.38.208:26656
E[2022-11-11|13:03:30.942][580372] dialing failed (attempts: 1): dial tcp 35.75.41.70:26656: i/o timeout. module=pex addr=83f7111062fd3d761077e800418be30767e69fd4@35.75.41.70:26656
E[2022-11-11|13:03:30.942][580372] dialing failed (attempts: 1): dial tcp 3.135.138.205:26656: i/o timeout. module=pex addr=bbd4aa19249ad42de4fe2ca99209894412fd4707@3.135.138.205:26656
E[2022-11-11|13:03:30.942][580372] dialing failed (attempts: 1): dial tcp 13.214.12.163:26656: i/o timeout. module=pex addr=add30aff52c2e43f071c7c2a8be797bef0ed8261@13.214.12.163:26656
E[2022-11-11|13:03:30.942][580372] dialing failed (attempts: 1): dial tcp 52.76.246.38:26656: i/o timeout. module=pex addr=e83abdf9222dd975dcc5b85d649fa8241797a9b4@52.76.246.38:26656
E[2022-11-11|13:03:30.942][580372] dialing failed (attempts: 1): dial tcp 3.64.37.17:26656: i/o timeout. module=pex addr=01b21d39f250a3a5411113aae4a7032eaf9b344e@3.64.37.17:26656
E[2022-11-11|13:03:30.942][580372] dialing failed (attempts: 1): dial tcp 54.255.93.228:26656: i/o timeout. module=pex addr=3449bb4d2180dfaa9ddb13776177b0e67f95ebb4@54.255.93.228:26656
E[2022-11-11|13:03:30.942][580372] dialing failed (attempts: 1): dial tcp 3.37.121.32:26656: i/o timeout. module=pex addr=833777703584393d763b60169e5ca204da91dd83@3.37.121.32:26656
E[2022-11-11|13:03:30.942][580372] dialing failed (attempts: 1): dial tcp 52.78.236.126:26656: i/o timeout. module=pex addr=54c195e08ff53e9fd31973dd73d530dcd1506807@52.78.236.126:26656
E[2022-11-11|13:03:31.167][580372] dialing failed (attempts: 1): dial tcp 46.137.225.25:26656: i/o timeout. module=pex addr=f13b2191fbb5b9f63acae4efb8f08bdc99a43992@46.137.225.25:26656
E[2022-11-11|13:03:31.167][580372] dialing failed (attempts: 1): dial tcp 54.151.129.9:26656: i/o timeout. module=pex addr=da32322e27dc9ef5002fed0416f05326fd27723f@54.151.129.9:26656
E[2022-11-11|13:03:31.193][580372] Stopping peer for error. module=p2p peer="Peer{MConn{175.41.191.69:26656} 7fa5b1d1f1e48659fa750b6aec702418a0e75f13 out}" err=EOF
E[2022-11-11|13:03:31.852][580372] Stopping peer for error. module=p2p peer="Peer{MConn{3.16.103.80:26656} e926c8154a2af4390de02303f0977802f15eafe2 out}" err=EOF
E[2022-11-11|13:03:31.859][580372] Stopping peer for error. module=p2p peer="Peer{MConn{35.74.8.189:26656} c8f32b793871b56a11d94336d9ce6472f893524b out}" err=EOF
E[2022-11-11|13:03:32.529][580372] dialing failed (attempts: 1): auth failure: secret conn failed: read tcp 192.168.1.135:57948->43.206.83.79:26656: i/o timeout. module=pex addr=acb4ce67f070e1039c5e9f30e41de15a648a3f77@43.206.83.79:26656
E[2022-11-11|13:03:32.619][580372] Stopping peer for error. module=p2p peer="Peer{MConn{3.16.103.80:26656} e926c8154a2af4390de02303f0977802f15eafe2 out}" err=EOF
E[2022-11-11|13:03:32.827][580372] Stopping peer for error. module=p2p peer="Peer{MConn{175.41.191.69:26656} 7fa5b1d1f1e48659fa750b6aec702418a0e75f13 out}" err=EOF
I[2022-11-11|13:03:32.870][580372] Height<15277001>, Tx<1>, BlockSize<5784>, RunTx<0ms>, . module=main 
panic: Failed to process committed block (15277001:31B4C3EBDADB751998CD16FD0B25C105AF4DBC5D1D831181FFEDEF492E9FFEC2): wrong Block.Header.AppHash.  Expected 84343AED828A775F06D3049D12CB71A09FB63FA149F4B36A245DC183637E2A6B, got 9D7D25D4977FAA0E7A08C5326FE169B17CD30FD8AC0F9A6FF20CD064F360EF58

goroutine 196 [running]:
github.com/okex/exchain/libs/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc000aa0000)
	/media/okex/exchain/libs/tendermint/blockchain/v0/reactor.go:394 +0x10fa
created by github.com/okex/exchain/libs/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
	/media/okex/exchain/libs/tendermint/blockchain/v0/reactor.go:133 +0x7a
cwbhhjl commented

try this with v1.6.5.1

  1. exchaind repair-state --home ${EXCHAIND_PATH} --start-height=15276900 --db_backend rocksdb
  2. start exchaind
dandavid3000 commented 
> 

2022/11/11 15:05:00 repair state latestBlockHeight = 15277000 	 commitVersion = 15276999
2022/11/11 15:05:01 repair state at version = 15276900
I[2022-11-11|15:05:02.902][590031] DeltaContext init. module=main uploadDelta=false downloadDelta=false buffer-size=0
I[2022-11-11|15:05:04.536][590031] Height<15276901>, Tx<1>, BlockSize<6265>, GasUsed<422078>, InvalidTxs<0>, RunTx<ApplyBlock<488ms>, abci<72ms>, saveResp<250ms>, persist<68ms>, mpUpdate<0ms>, saveState<96ms>>, Iavl[getnode<8921>, rdb<8921>, rdbTs<138ms>, savenode<818>], DeliverTxs[RunAnte<2ms>, RunMsg<27ms>, refund<0ms>]. module=main 
2022/11/11 15:05:04 Repaired block height 15276901
2022/11/11 15:05:04 Repaired app hash 6ED5971AB38E9F7A5B0D80BA7D1311691CA00534DECE3FB50C2FA1FAEE2A50DC
I[2022-11-11|15:05:06.185][590031] Height<15276902>, Tx<1>, BlockSize<2853>, GasUsed<57212>, InvalidTxs<0>, RunTx<ApplyBlock<490ms>, abci<54ms>, saveResp<310ms>, persist<59ms>, mpUpdate<0ms>, saveState<66ms>>, Iavl[getnode<6778>, rdb<6778>, rdbTs<76ms>, savenode<719>], DeliverTxs[RunAnte<0ms>, RunMsg<12ms>, refund<0ms>]. module=main 
2022/11/11 15:05:06 Repaired block height 15276902
2022/11/11 15:05:06 Repaired app hash 5559491C3C1B7C1704ABDC34C9E70A6A949FE16796F2144974241D4AE00658E3
I[2022-11-11|15:05:08.163][590031] Height<15276903>, Tx<3>, BlockSize<4305>, GasUsed<805883>, InvalidTxs<0>, RunTx<ApplyBlock<464ms>, abci<42ms>, saveResp<256ms>, persist<38ms>, mpUpdate<0ms>, saveState<126ms>>, Iavl[getnode<11831>, rdb<11831>, rdbTs<75ms>, savenode<1152>], DeliverTxs[RunAnte<0ms>, RunMsg<26ms>, refund<0ms>]. module=main 
2022/11/11 15:05:08 Repaired block height 15276903
2022/11/11 15:05:08 Repaired app hash DE183DD2CD8FE1CF6F4DA0DCA28CBFCEF7778B9044170C0A61C8FD5B44616CFE
panic: Cannot call IncrementProposerPriority with non-positive times

goroutine 1 [running]:
github.com/okex/exchain/libs/tendermint/types.(*ValidatorSet).IncrementProposerPriority(0x2681488?, 0xc0005169e0?)
	/media//okex/exchain/libs/tendermint/types/validator_set.go:94 +0xe5
github.com/okex/exchain/libs/tendermint/state.LoadValidators({0x2681488, 0xc0005169e0}, 0xe91b67)
	/media/okex/exchain/libs/tendermint/state/store.go:471 +0xb5
github.com/okex/exchain/libs/tendermint/state.getBeginBlockValidatorInfo(0xc0001a6000, {0x2681488, 0xc0005169e0})
	/media/okex/exchain/libs/tendermint/state/execution.go:571 +0xd2
github.com/okex/exchain/libs/tendermint/state.execBlockOnProxyApp(0xc00150c1f8)
	/media/okex/exchain/libs/tendermint/state/execution.go:481 +0x266
github.com/okex/exchain/libs/tendermint/state.(*BlockExecutor).runAbci(0xc0007ac750, 0xc0001a6000, 0xc00150c2b8?)
	/media/okex/exchain/libs/tendermint/state/execution.go:348 +0x145
github.com/okex/exchain/libs/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc000b920a0, 0x6}}, {0xc000b920b0, 0xa}, 0xe91b67, {{0xc0014ca420, 0x20, ...}, ...}, ...}, ...)
	/media/okex/exchain/libs/tendermint/state/execution.go:237 +0x572
github.com/okex/exchain/libs/tendermint/state.(*BlockExecutor).ApplyBlockWithTrace(_, {{{0xb, 0x0}, {0xc000b920a0, 0x6}}, {0xc000b920b0, 0xa}, 0xe91b67, {{0xc0014ca420, 0x20, ...}, ...}, ...}, ...)
	/media/okex/exchain/libs/tendermint/state/execution.go:318 +0x110
github.com/okex/exchain/app.doRepair(_, {{{0xb, 0x0}, {0xc000b920a0, 0x6}}, {0xc000b920b0, 0xa}, 0xe91b67, {{0xc0014ca420, 0x20, ...}, ...}, ...}, ...)
	/media/okex/exchain/app/repair_state.go:203 +0x7fb
github.com/okex/exchain/app.RepairState(0xc0005a4798, 0x0)
	/media/okex/exchain/app/repair_state.go:142 +0x705
main.repairStateCmd.func2(0xc0001e7340?, {0xc000c679f0?, 0x0?, 0x5?})
	/media/okex/exchain/cmd/exchaind/repair_data.go:29 +0x5d
github.com/spf13/cobra.(*Command).execute(0xc0001e7340, {0xc000c679a0, 0x5, 0x5})
	/home/dvo/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:854 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0xc00092c000)
	/home/dvo/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:958 +0x39d
github.com/spf13/cobra.(*Command).Execute(...)
	/home/dvo/go/pkg/mod/github.com/spf13/cobra@v1.1.1/command.go:895
github.com/okex/exchain/libs/tendermint/libs/cli.Executor.Execute({0xc00092c000?, 0x238eba0?})
	/media/okex/exchain/libs/tendermint/libs/cli/setup.go:89 +0x36
main.main()
	/media/okex/exchain/cmd/exchaind/main.go:114 +0x7eb

After that I rerun the node

. module=main 
panic: StoreBlockHeight (15277000) > StateBlockHeight + 1 (15276904)

goroutine 1 [running]:
github.com/okex/exchain/libs/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc001292e00, 0x6}}, {0xc001292e10, 0xa}, 0xe91b67, {{0xc00127c0a0, 0x20, ...}, ...}, ...}, ...)
cwbhhjl commented

Please try to use the old version to execute the repair-state command and see if it still panics.

  • 👍1
dandavid3000 commented

Please try to use the old version to execute the repair-state command and see if it still panics.

Thanks