The number of unprocessed events in channel `polkadot-network-bridge` exceeded 100000.
Opened this issue · 0 comments
ioannist commented
Related to:
paritytech/polkadot#6904
Moonbeam fullnode 0.33 terminates when under heavy query load (for historical blocks) for 10-20 minutes.
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:32 [Relaychain] The number of unprocessed events in channel `polkadot-network-bridge` exceeded 100000.
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: The channel was created at:
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 0: sc_network::service::out_events::channel
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 1: <alloc::sync::Arc<sc_network::service::NetworkService<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::OpaqueExtrinsic>,primitive_types::H256>> as polkadot_network_bridge::network::Network>::event_stream
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 2: polkadot_overseer::spawn
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 3: polkadot_overseer::OverseerBuilder<polkadot_overseer::Init<S>,polkadot_overseer::Init<CandidateValidation>,polkadot_overseer::Init<PvfChecker>,polkadot_overseer::Init<CandidateBacking>,polkadot_overseer::Init<StatementDistribution>,polkadot_overseer::Init<AvailabilityDistribution>,polkadot_overseer::Init<AvailabilityRecovery>,polkadot_overseer::Init<BitfieldSigning>,polkadot_overseer::Init<BitfieldDistribution>,polkadot_overseer::Init<Provisioner>,polkadot_overseer::Init<RuntimeApi>,polkadot_overseer::Init<AvailabilityStore>,polkadot_overseer::Init<NetworkBridgeRx>,polkadot_overseer::Init<NetworkBridgeTx>,polkadot_overseer::Init<ChainApi>,polkadot_overseer::Init<CollationGeneration>,polkadot_overseer::Init<CollatorProtocol>,polkadot_overseer::Init<ApprovalDistribution>,polkadot_overseer::Init<ApprovalVoting>,polkadot_overseer::Init<GossipSupport>,polkadot_overseer::Init<DisputeCoordinator>,polkadot_overseer::Init<DisputeDistribution>,polkadot_overseer::Init<ChainSelection>,polkadot_overseer::Init<std::collections::hash::map::HashMap<primitive_types::H256,alloc::vec::Vec<futures_channel::oneshot::Sender<core::result::Result<(),polkadot_node_subsystem_types::errors::SubsystemError>>>>>,polkadot_overseer::Init<std::collections::hash::map::HashMap<primitive_types::H256,alloc::sync::Arc<polkadot_node_jaeger::spans::Span>>>,polkadot_overseer::Init<std::collections::hash::map::HashMap<primitive_types::H256,u32>>,polkadot_overseer::Init<SupportsParachains>,polkadot_overseer::Init<lru::LruCache<primitive_types::H256,()>>,polkadot_overseer::Init<polkadot_overseer::metrics::Metrics>>::build_with_connector
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 4: <polkadot_service::overseer::RealOverseerGen as polkadot_service::overseer::OverseerGen>::generate
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 5: polkadot_service::new_full
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 6: moonbeam_service::build_relay_chain_interface::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 7: moonbeam_service::start_node_impl::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 8: moonbeam_cli::command::run::{{closure}}::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 9: sc_cli::runner::Runner<C>::run_node_until_exit
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 10: moonbeam_cli::command::run
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 11: moonbeam::main
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 12: std::sys_common::backtrace::__rust_begin_short_backtrace
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 13: std::rt::lang_start::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 14: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/ops/function.rs:284:13
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 15: std::panicking::try::do_call
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panicking.rs:500:40
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 16: std::panicking::try
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panicking.rs:464:19
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 17: std::panic::catch_unwind
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panic.rs:142:14
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 18: std::rt::lang_start_internal::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/rt.rs:148:48
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 19: std::panicking::try::do_call
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panicking.rs:500:40
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 20: std::panicking::try
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panicking.rs:464:19
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 21: std::panic::catch_unwind
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panic.rs:142:14
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 22: std::rt::lang_start_internal
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/rt.rs:148:20
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 23: main
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 24: <unknown>
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 25: __libc_start_main
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 26: _start
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: The last event was sent from:
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 0: sc_network::service::out_events::OutChannels::send::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 1: sc_network::service::out_events::OutChannels::send
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 2: sc_network::service::NetworkWorker<B,H>::handle_swarm_event
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 3: sc_network::service::NetworkWorker<B,H>::next_action::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 4: <core::pin::Pin<P> as core::future::future::Future>::poll
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 5: sc_service::build_network_future::{{closure}}::{{closure}}::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 6: sc_service::builder::build_network::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 7: tokio::runtime::task::raw::poll
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 8: std::sys_common::backtrace::__rust_begin_short_backtrace
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 9: core::ops::function::FnOnce::call_once{{vtable.shim}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 10: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/alloc/src/boxed.rs:1985:9
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 11: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/alloc/src/boxed.rs:1985:9
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 12: std::sys::unix::thread::Thread::new::thread_start
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/sys/unix/thread.rs:108:17
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 13: <unknown>
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 14: <unknown>
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:
Oct 30 12:27:33 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:33 [Relaychain] ✨ Imported #17948289 (0xba0d…3746)
Oct 30 12:27:36 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:36 [Relaychain] 💤 Idle (14 peers), best: #17948289 (0xba0d…3746), finalized #17948286 (0x115e…8a1a), ⬇ 663.0kiB/s ⬆ 9.8kiB/s
Oct 30 12:27:36 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:36 [🌗] ⚙️ Preparing 0.0 bps, target=#4759845 (3 peers), best: #4759823 (0xf164…3409), finalized #4759803 (0x4f41…067e), ⬇ 5.8kiB/s ⬆ 0.7kiB/s
Oct 30 12:27:41 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:41 [Relaychain] 💤 Idle (18 peers), best: #17948289 (0xba0d…3746), finalized #17948287 (0x6224…01c5), ⬇ 116.7kiB/s ⬆ 7.3kiB/s
Oct 30 12:27:41 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:41 [🌗] ⚙️ Preparing 0.0 bps, target=#4759845 (3 peers), best: #4759823 (0xf164…3409), finalized #4759803 (0x4f41…067e), ⬇ 9.4kiB/s ⬆ 0.1kiB/s
Oct 30 12:27:42 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:42 [Relaychain] cannot query the runtime API version: Api called for an unknown Block: State already discarded for 0x844abf4aa5868e67ba4aba2d97571d55976183ddc10da872c4de94abd1773126
Oct 30 12:27:42 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:42 [Relaychain] ✨ Imported #17948290 (0x0e02…9936)
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] error receiving message from subsystem context: Generated(Context("Signal channel is terminated and empty.")) err=Generated(Context("Signal channel is terminated and empty."))
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] subsystem exited with error subsystem="availability-recovery-subsystem" err=FromOrigin { origin: "availability-recovery", source: Generated(Context("Signal channel is terminated and empty.")) }
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] subsystem exited with error subsystem="provisioner-subsystem" err=FromOrigin { origin: "provisioner", source: OverseerExited(Generated(Context("Signal channel is terminated and empty."))) }
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] subsystem exited with error subsystem="network-bridge-rx-subsystem" err=FromOrigin { origin: "network-bridge", source: SubsystemError(Generated(Context("Signal channel is terminated and empty."))) }
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] subsystem exited with error subsystem="network-bridge-tx-subsystem" err=FromOrigin { origin: "network-bridge", source: SubsystemError(Generated(Context("Signal channel is terminated and empty."))) }
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] err=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] Overseer exited with error err=Generated(SubsystemStalled("approval-distribution-subsystem"))
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] subsystem exited with error subsystem="chain-api-subsystem" err=FromOrigin { origin: "chain-api", source: Generated(Context("Signal channel is terminated and empty.")) }
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] subsystem exited with error subsystem="bitfield-signing-subsystem" err=FromOrigin { origin: "bitfield-signing", source: Generated(Context("Signal channel is terminated and empty.")) }
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] Essential task `overseer` failed. Shutting down service.
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] subsystem exited with error subsystem="candidate-validation-subsystem" err=FromOrigin { origin: "candidate-validation", source: Generated(Context("Signal channel is terminated and empty.")) }
Oct 30 12:27:57 stakebaby-chalandri moonbeam[327057]: Error: Service(Other("Essential task failed."))
Oct 30 12:27:57 stakebaby-chalandri systemd[1]: moonbeam.service: Main process exited, code=exited, status=1/FAILURE
Oct 30 12:27:57 stakebaby-chalandri systemd[1]: moonbeam.service: Failed with result 'exit-code'.
Runtime settings:
ExecStart=/var/lib/moonbeam-data/moonbeam-skylake \
--execution wasm \
--wasm-execution compiled \
--state-pruning=archive \
--port 30343 \
--rpc-port 9943 \
--db-cache 4000 \
--base-path /ssd-pool/chains/moonbeam \
--chain moonbeam \
--max-runtime-instances 256 \
--trie-cache-size 573741824 \
--rpc-max-request-size 100 \
--rpc-max-response-size 100 \
--rpc-max-connections 300 \
--in-peers 3 \
--out-peers 3 \
--max-parallel-downloads 2 \
--no-hardware-benchmarks \
--no-grandpa \
--no-telemetry \
--no-prometheus \