Nodeos stuck and doesn't want to stop

[fenidik]

Having issue with nodeos version 2.1.0-1
os ubuntu-20.04

Process is running and writing output. But doesn't answer on api requests and doesn't want to stop. Kill does nothing with process.

Strace output

strace -p 627177
strace: Process 627177 attached
futex(0x55c5c5789b18, FUTEX_WAIT_PRIVATE, 0, NULL

GDB backtrace

#0  0x00007f37f0afc376 in futex_wait (private=<optimized out>, expected=0, futex_word=0x55c5c5789b18)
    at ../sysdeps/nptl/futex-internal.h:141
#1  futex_wait_simple (private=<optimized out>, expected=0, futex_word=0x55c5c5789b18)
    at ../sysdeps/nptl/futex-internal.h:172
#2  __condvar_quiesce_and_switch_g1 (private=<optimized out>, g1index=<synthetic pointer>, wseq=<optimized out>, cond=0x0)
    at pthread_cond_common.c:416
#3  __pthread_cond_broadcast (cond=0x0) at pthread_cond_broadcast.c:73
#4  0x000055c5c5789af0 in ?? ()
#5  0x000055c5c5789ac8 in ?? ()
#6  0xffffff3700000000 in ?? ()
#7  0xffffff3767cf70e8 in ?? ()
#8  0xc19791c669947200 in ?? ()
#9  0x00007ffc87e59450 in ?? ()
#10 0x00007ffc87e59398 in ?? ()
#11 0x000055c5c5789ab0 in ?? ()
#12 0x000055c5c5789af0 in ?? ()
#13 0x00007ffc87e59950 in ?? ()
#14 0x00007ffc87e59950 in ?? ()
#15 0x00007ffc87e59c90 in ?? ()
#16 0x000055c5be93e126 in std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) ()
#17 0x000055c5be93ece3 in std::__1::__assoc_sub_state::__sub_wait(std::__1::unique_lock<std::__1::mutex>&) ()
#18 0x000055c5bcfc525d in std::__1::__assoc_state<std::__1::shared_ptr<eosio::chain::block_state> >::move() ()
#19 0x000055c5bcee6edc in eosio::chain::controller_impl::push_block(std::__1::future<std::__1::shared_ptr<eosio::chain::block_state> >&, std::__1::function<void (std::__1::vector<std::__1::shared_ptr<eosio::chain::block_state>, std::__1::allocator<std::__1::shared_ptr<eosio::chain::block_state> > > const&)> const&, std::__1::function<std::__1::shared_ptr<eosio::chain::transaction_metadata> (fc::sha256 const&)> const&) ()
#20 0x000055c5bcec4f30 in eosio::chain::controller::push_block(std::__1::future<std::__1::shared_ptr<eosio::chain::block_state> >&, std::__1::function<void (std::__1::vector<std::__1::shared_ptr<eosio::chain::block_state>, std::__1::allocator<std::__1::shared_ptr<eosio::chain::block_state> > > const&)> const&, std::__1::function<std::__1::shared_ptr<eosio::chain::transaction_metadata> (fc::sha256 const&)> const&) ()
#21 0x000055c5bc970619 in eosio::producer_plugin_impl::on_incoming_block(std::__1::shared_ptr<eosio::chain::signed_block> const&, std::__1::optional<fc::sha256> const&) ()
#22 0x000055c5bc790950 in boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, std::__1::shared_ptr<eosio::chain::signed_block> const&, std::__1::optional<fc::sha256> const&>, std::__1::__list_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::__1::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__1::shared_ptr<eosio::chain::signed_block> const&, std::__1::optional<fc::sha256> const&), boost::function<bool (std::__1::shared_ptr<eosio::chain::signed_block> const&, std::__1::optional<fc::sha256> const&)> >, boost::signals2::mutex> >, void*>, boost::signals2::detail::connection_body<std::__1::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (std::__1::shared_ptr<eosio::chain::signed_block> const&, std::__1::optional<fc::sha256> const&), boost::function<bool (std::__1::shared_ptr<eosio::chain::signed_block> const&, std::__1::optional<fc::sha256> const&)> >, boost::signals2::mutex> >::dereference() const ()
#23 0x000055c5bc7900d2 in boost::signals2::detail::signal_impl<bool (std::__1::shared_ptr<eosio::chain::signed_block> const&, std::__1::optional<fc::sha256> const&), appbase::first_provider_policy<bool (std::__1::shared_ptr<eosio::chain::signed_block> const&, std::__1::optional<fc::sha256> const&)>, int, std::__1::less<int>, boost::function<bool (std::__1::shared_ptr<eosio::chain::signed_block> const&, std::__1::optional<fc::sha256> const&)>, boost::function<bool (boost::signals2::connection const&, std::__1::shared_ptr<eosio::chain::signed_block> const&, std::__1::optional<fc::sha256> const&)>, boost::signals2::mutex>::operator()(std::__1::shared_ptr<eosio::chain::signed_block> const&, std::__1::optional<fc::sha256> const&) ()
#24 0x000055c5bc6c7bde in eosio::chain_plugin::accept_block(std::__1::shared_ptr<eosio::chain::signed_block> const&, fc::sha256 const&) ()
#25 0x000055c5bc4e3ae1 in eosio::connection::process_signed_block(fc::sha256 const&, std::__1::shared_ptr<eosio::chain::signed_block>) ()
#26 0x000055c5bc513877 in appbase::execution_priority_queue::queued_handler<boost::asio::executor_binder<eosio::connection::handle_message(fc::sha256 const&, std::__1::shared_ptr<eosio::chain::signed_block>)::$_17, appbase::execution_priority_queue--Type <RET> for more, q to quit, c to continue without paging--
::executor> >::execute() ()
#27 0x000055c5bc18f538 in appbase::application::exec() ()
#28 0x000055c5bc183080 in main ()

What's wrong with it?

[heifner]

Can you provide the backtrace of all the threads?

[fenidik]

Unfortunately no. Have to kill -9 this process.

[heifner]

If it happens again, please provide backtrace from all the threads. It is stuck on a mutex and need to see the other threads to know what is holding the mutex and stopping it.

[fenidik]

Deal. Will reopen if happens again