[ld:admin] NodeID.h:53] index() Check failed: isNodeID() Caught coredump signal 6
zhengxiaochuan-3 opened this issue · 2 comments
zhengxiaochuan-3 commented
After run a cmd select * from info_config on ldshell , all my four nodes crash down
I0911 14:05:33.984902 39 [ld:s0:db-bg-fl] PartitionedRocksDBStore.cpp:6386] flushBackgroundThreadRun() Flushing: []. Total stats: active: 0.001MB in 1 memtables, flushing: 0.000MB, pinned: 0.000MB. Eval time: 0ms, flush init: 0ms, throttle eval: 0ms, npartitions: 8. Existing memtables: metadata A:0.001MB, []
C0911 14:05:38.533110 12 [ld:admin] NodeID.h:53] index() Check failed: isNodeID()
handle_fatal_signal(): Caught coredump signal 6
I0911 14:05:39.467410 81 [ld:srv:WG30] Socket.cpp:1052] close() Closing socket C116(10.35.98.12:14178). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467462 55 [ld:srv:WG4] Socket.cpp:1052] close() Closing socket C90(10.35.98.12:13640). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467620 83 [ld:srv:WF0] Socket.cpp:1052] close() Closing socket N0:1(10.35.98.12:16112). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467662 81 [ld:srv:WG30] Socket.cpp:1052] close() Closing socket N0:1(10.35.98.12:16111). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467686 51 [ld:srv:WG0] Socket.cpp:1052] close() Closing socket N0:1(10.35.98.12:16111). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467705 77 [ld:srv:WG26] Socket.cpp:1052] close() Closing socket C3(10.35.98.12:64316). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467729 59 [ld:srv:WG8] Socket.cpp:1052] close() Closing socket N0:1(10.35.98.12:16111). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467746 83 [ld:srv:WF0] Socket.cpp:1052] close() Closing socket C1(10.35.98.12:22200). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467764 73 [ld:srv:WG22] Socket.cpp:1052] close() Closing socket C6(10.35.98.12:64340). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467803 55 [ld:srv:WG4] Socket.cpp:1052] close() Closing socket C8(10.35.98.12:64350). Reason: PEER_CLOSED: connection closed by peer
@ 000055fe6aa25d9c facebook::logdevice::handle_fatal_signal(int)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6a0fe613 coredump_signal_handler(int)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 00007f8b07b2988f (unknown)
@ 00007f8b03d15e97 gsignal
@ 00007f8b03d17800 abort
@ 000055fe6b419c64 facebook::logdevice::dbg::ld_check_fail_impl(facebook::logdevice::dbg::CheckType, char const*, char const*, char const*, int)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6a1683ce facebook::logdevice::NodeID::index() const
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6a207b44 facebook::logdevice::commands::InfoConfig::metadata(facebook::logdevice::Configuration const&)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6a2078db facebook::logdevice::commands::InfoConfig::run()
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6a1f7eff facebook::logdevice::CommandProcessor::processCommand(char const*, folly::SocketAddress const&)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6a1f16d4 facebook::logdevice::CommandListener::processCommand(bufferevent*, char const*, bool, facebook::logdevice::CommandListener::ConnectionState*)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6a1f0195 facebook::logdevice::CommandListener::readCallback(bufferevent*, void*)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 00007f8b06d3ae31 (unknown)
@ 00007f8b06d408f7 (unknown)
@ 00007f8b06d4133e event_base_loop
@ 000055fe6b660d4f facebook::logdevice::EvBase::loop()
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6b1408a6 facebook::logdevice::EventLoop::run()
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6b13fd25 facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}::operator()() const
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6b140da0 void std::__invoke_impl<void, facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>(std::__invoke_other, facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}&&)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6b140a66 std::__invoke_result<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>::type std::__invoke<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>(std::__invoke_result&&, (facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}&&)...)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6b1412b3 decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6b141284 std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> >::operator()()
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6b141263 std::thread::_State_impl<std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> > >::_M_run()
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 00007f8b0473b66e (unknown)
@ 00007f8b07b1e6da start_thread
@ 00007f8b03df888e clone
handle_fatal_signal(): processed segments: 209
handle_fatal_signal(): unmapped pages - unmap calls: 283 - 12
MohamedBassem commented
The crash at this line: https://github.com/facebookincubator/LogDevice/blob/master/logdevice/server/admincommands/InfoConfig.h#L72
This is a bug because if the source of the config is the node itself, getServerOrigin() will return an invalid NodeID, and calling index() on it will crash the server. Thanks for the report, that's quite a serious bug :)
Will send a fix for it today.
MohamedBassem commented
Just landed the fix. Thanks again for the report!