Consensus failure on block 11860000 after upgrade to v2.12.10

Question

Consensus failure on block 11860000 after upgrade to v2.12.10

ttibord opened this issue 4 months ago · 3 comments

Our nodes running version 2.11.8, with halt-height=0, stopped syncing today at:

{
  "latest_block_hash": "9CBA463E04D108A07AD9E4C47349F05EF3454CACC8A06D052D6AC26784721453",
  "latest_app_hash": "89415A0EE482DD716A7F565381DD03168FB8EE955B2AB824A7DE1F6BFE3B4821",
  "latest_block_height": "11860000",
  "latest_block_time": "2024-08-30T08:09:12.886506586Z",
  "earliest_block_hash": "5C077EFA10EFECC044A4861860857262CB077AAA9E26CE9D05BB5A8ABC8602D6",
  "earliest_app_hash": "D1007B693F71E335859941F791CAECA6CDD3D74AD30C4E332EBBE445E7277126",
  "earliest_block_height": "11605097",
  "earliest_block_time": "2024-08-12T13:58:52.447594413Z",
  "catching_up": false
}

After restart node crashes, reporting needing upgrade to 2.12.0, marked as pre-release at time of posting:

1:00PM INF Replay last block using real app module=consensus
1:00PM ERR UPGRADE "v2.12" NEEDED at height: 11860000: {"binaries": {"linux/amd64": "https://github.com/terra-money/core/releases/download/v2.12.4/terra_2.12.4_Linux_x86_64.tar.gz?checksum=sha256:82b58db8e14e052ba418f42ac0028d620e640780f3181b2d11c0c8ae804b04a0"}} module=server
panic: UPGRADE "v2.12" NEEDED at height: 11860000: {"binaries": {"linux/amd64": "https://github.com/terra-money/core/releases/download/v2.12.4/terra_2.12.4_Linux_x86_64.tar.gz?checksum=sha256:82b58db8e14e052ba418f42ac0028d620e640780f3181b2d11c0c8ae804b04a0"}}
goroutine 133 [running]:
github.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker(_, {{0x45bbb58, 0xc0000780c0}, {0x45d3350, 0xc059a4a540}, {{0xb, 0x0}, {0xc0599e4530, 0x9}, 0xb4f820, ...}, ...}, ...)
        github.com/cosmos/cosmos-sdk@v0.47.10/x/upgrade/abci.go:78 +0xe54

After upgrade node no longer syncs reporting CONSENSUS FAILURE (last line in logs below):

1:09PM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
1:09PM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service"
1:09PM INF ABCI Handshake App Info hash="�AZ\x0e���qj\x7fVS��\x03\x16����[*�$��\x1fk�;H!" height=11859999 module=consensus protocol-version=0 software-version=2.12.0
1:09PM INF ABCI Replay Blocks appHeight=11859999 module=consensus stateHeight=11859999 storeHeight=11860000
1:09PM INF Replay last block using real app module=consensus
1:09PM INF applying upgrade "v2.12" at height: 11860000 module=server
1:09PM INF burned tokens from module account amount=2839045000000uluna from=not_bonded_tokens_pool module=x/bank
1:09PM INF burned tokens from module account amount=48ibc/2C962DAB9F57FE0921435426AE75196009FAA1981BF86991203C8411F8980FDB from=not_bonded_tokens_pool module=x/bank
1:09PM INF burned tokens from module account amount=41503400000000000000ibc/644A3695F405A67B0DD6E414602B836614BA3566ABBF4F2DA6ED0D2BB4332125 from=not_bonded_tokens_pool module=x/bank
1:09PM INF burned tokens from module account amount=212ibc/B3504E092456BA618CC28AC671A71FB08C6CA0FD0BE7C8A5B5A3E2DD933CC9E4 from=not_bonded_tokens_pool module=x/bank
...
1:13PM INF Catchup by replaying consensus messages height=11860001 module=consensus
1:13PM INF Replay: Done module=consensus
1:13PM INF Timed out dur=-26187.058192 height=11860001 module=consensus round=0 step=RoundStepNewHeight
1:13PM INF Timed out dur=3000 height=11860001 module=consensus round=0 step=RoundStepPropose
1:13PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=1 numToDial=9
1:13PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=1 numToDial=9
1:13PM INF service start impl="Peer{MConn{188.40.71.173:11756} d33aafd689aa5fdc72c90684de86b3945d48039e out}" module=p2p msg="Starting Peer service" peer=d33aafd689aa5fdc72c90684de86b3945d48039e@188.40.71.173:11756
1:13PM INF service start impl=MConn{188.40.71.173:11756} module=p2p msg="Starting MConnection service" peer=d33aafd689aa5fdc72c90684de86b3945d48039e@188.40.71.173:11756
1:13PM INF resetting proposal info height=11860001 module=consensus proposer=68D02947EB450DCB13EF0406C90AFF70225116F6 round=1
1:13PM INF commit is for a block we do not know about; set ProposalBlock=nil commit=895D9D9858BC91890ECCE5D809F7161ADFEB34EE4C847CC46931F96D894BC2BD commit_round=1 height=11860001 module=consensus proposal=
1:13PM INF received complete proposal block hash=895D9D9858BC91890ECCE5D809F7161ADFEB34EE4C847CC46931F96D894BC2BD height=11860001 module=consensus
1:13PM ERR CONSENSUS FAILURE!!! err="+2/3 committed an invalid block: wrong Block.Header.AppHash.  Expected FEDC250E74237B0C751C8F4E03D16D335CAA2D9D9314D12C9CD20D42D2480CE0, got 1F9F7DE43473B8D18B294B4A529155C5097F6959D862E22537230ABB7BAA0552" module=consensus stack="goroutine 346 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/cometbft/cometbft@v0.37.4/consensus/state.go:736 +0x4c\npanic({0x3378e60, 0xc27eb46420})\n\truntime/panic.go:884 +0x213\ngithub.com/cometbft/cometbft/consensus.(*State).finalizeCommit(0xc00203a380, 0xb4f821)\n\tgithub.com/cometbft/cometbft@v0.37.4/consensus/state.go:1650 +0x101c\ngithub.com/cometbft/cometbft/consensus.(*State).tryFinalizeCommit(0xc00203a380, 0xb4f821)\n\tgithub.com/cometbft/cometbft@v0.37.4/consensus/state.go:1619 +0x2ff\ngithub.com/cometbft/cometbft/consensus.(*State).handleCompleteProposal(0xc00203a380, 0xc27eb51000?)\n\tgithub.com/cometbft/cometbft@v0.37.4/consensus/state.go:2006 +0x399\ngithub.com/cometbft/cometbft/consensus.(*State).handleMsg(0xc00203a380, {{0x4592520, 0xc27ea71ba8}, {0xc27e96a5a0, 0x28}})\n\tgithub.com/cometbft/cometbft@v0.37.4/consensus/state.go:846 +0x1b7\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine(0xc00203a380, 0x0)\n\tgithub.com/cometbft/cometbft@v0.37.4/consensus/state.go:772 +0x3f0\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart\n\tgithub.com/cometbft/cometbft@v0.37.4/consensus/state.go:383 +0x12d\n"

We have node snapshots from this morning, couple of hours prior to the halting block, so happy to roll back and sync with necessary versions if nodes can't be repaired in current state.

Thank you!

Answer 1 · 2024-09-01T04:59:50.000Z

Hi! You should be updating to https://github.com/terra-money/core/releases/tag/v2.12.4 instead as per the governance proposal here: https://station.terra.money/proposal/phoenix-1/4818

Answer 2 · 2024-09-01T06:49:25.000Z

@javiersuweijie on upgrade to 2.12.4, we are getting:

6:42AM ERR Error in validation err="wrong Block.Header.AppHash.  Expected FEDC250E74237B0C751C8F4E03D16D335CAA2D9D9314D12C9CD20D42D2480CE0, got 1F9F7DE43473B8D18B294B4A529155C5097F6959D862E22537230ABB7BAA0552" module=blockchain
6:42AM ERR Stopping peer for error err="Reactor validation error: wrong Block.Header.AppHash.  Expected FEDC250E74237B0C751C8F4E03D16D335CAA2D9D9314D12C9CD20D42D2480CE0, got 1F9F7DE43473B8D18B294B4A529155C5097F6959D862E22537230ABB7BAA0552" module=p2p peer="Peer{MConn{37.27.0.251:26656} 073f2d18f8cf91fba29b16790fb38f71c18dd983 out}"

What version and halting block should have worked prior to the upgrade block?

Answer 3 · 2024-09-01T10:27:47.000Z

Nevermind, synced again on 2.11.8. to a block before the halting one, let it reach the halting block, and now 2.12.4 works, thank you!