Node (v0.32.1) crashloop: tokio-runtime-worker => Database missing expected key
Closed this issue · 1 comments
SimonKraus commented
Hi team,
Hetavalidation was finding their backup node in a crashloop with the stack trace below.
The node was running v0.32.1 and after upgrading it to 0.33.0 the problem was resolved and the node was able to catchup and sync again without the need of a database revert.
As the problem just happened with an outdated client version I'm sure it can be closed directly - in case needed I persisted the full logs from the last 24 hours.
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: ====================
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: Version: 0.32.1-6ed87ceeb65
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 0: sp_panic_handler::set::{{closure}}
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: at rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/alloc/src/boxed.rs:2001:9
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: std::panicking::rust_panic_with_hook
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: at rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:696:13
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 2: std::panicking::begin_panic_handler::{{closure}}
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: at rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:583:13
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 3: std::sys_common::backtrace::__rust_end_short_backtrace
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: at rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:150:18
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 4: rust_begin_unwind
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: at rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:579:5
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 5: core::panicking::panic_fmt
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: at rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:64:14
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 6: core::result::unwrap_failed
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: at rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/result.rs:1750:5
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 7: <sp_state_machine::ext::Ext<H,B> as sp_externalities::Externalities>::storage
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 8: sp_io::storage::get_version_1
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 9: sp_io::storage::ExtStorageGetVersion1::call
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 10: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1),R>>::into_func::wasm_to_host_shim
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 11: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 12: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 13: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 14: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 15: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 16: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 17: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 18: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 19: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 20: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 21: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 22: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 23: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 24: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 25: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 26: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 27: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 28: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 29: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 30: <unknown>
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 31: wasmtime_runtime::traphandlers::catch_traps::call_closure
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 32: wasmtime_setjmp
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 33: sc_executor_wasmtime::runtime::perform_call
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 34: <sc_executor_wasmtime::runtime::WasmtimeInstance as sc_executor_common::wasm_runtime::WasmInstance>::call_with_allocation_stats
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 35: sc_executor_common::wasm_runtime::WasmInstance::call_export
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 36: sc_executor::native_executor::WasmExecutor<H>::with_instance::{{closure}}
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 37: <sc_executor::native_executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 38: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 39: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_using_consensus_failure_handler
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 40: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 41: <moonbeam_runtime::RuntimeApiImpl<__SrApiBlock__,RuntimeApiImplCall> as sp_api::Core<__SrApiBlock__>>::__runtime_api_internal_call_api_at
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 42: <&sc_service::client::client::Client<B,E,Block,RA> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 43: <alloc::sync::Arc<T> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 44: <fc_consensus::FrontierBlockImport<B,I,C> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 45: <nimbus_consensus::import_queue::NimbusBlockImport<I> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 46: <alloc::boxed::Box<dyn sc_consensus::block_import::BlockImport<B>+Transaction = Transaction+Error = sp_consensus::error::Error+core::marker::Send+core::marker::Sync> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 47: sc_consensus::import_queue::basic_queue::BlockImportWorker<B>::new::{{closure}}
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 48: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 49: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 50: tokio::runtime::task::raw::poll
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 51: std::sys_common::backtrace::__rust_begin_short_backtrace
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 52: core::ops::function::FnOnce::call_once{{vtable.shim}}
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 53: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: at rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/alloc/src/boxed.rs:1987:9
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: at rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/alloc/src/boxed.rs:1987:9
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: std::sys::unix::thread::Thread::new::thread_start
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: at rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys/unix/thread.rs:108:17
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 54: start_thread
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: 55: clone
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: Thread 'tokio-runtime-worker' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0xbb6147b5354458e6a0a063579f6e172ee1815d92d8a7384fd38e1fccb05a430f"', /root/.cargo/git/checkouts/substrate-189071a041b0d328/e7de773/primitives/state-machine/src/ext.rs:192
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: This is a bug. Please report it at:
Oct 26 13:27:07 Ubuntu-2004-focal-64-minimal moonbeam[3931]: https://github.com/PureStake/moonbeam/issues/new
crystalin commented
Thank you for reporting it, as the issue is not appearing in v0.33.0 I'm going to close it, but we will investigate internally :)