gNB crashing for long run
vvdn5gtest opened this issue · 4 comments
Issue Description
While testing the main branch(hash f3ed07a) with latest commit hash the gNB crashed after 20 mins. The test setup is in C1 topology and crash occured while pushing data from UPF.Attaching crash log for your reference.
Stopping .. Stack trace (most recent call last) in thread 6909: #17 Object "", at 0xffffffffffffffff, in #16 Source "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3 [0x7ff02728784f] #15 Source "./nptl/pthread_create.c", line 442, in start_thread [0x7ff0271f5ac2] #14 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7ff02756f252, in #13 Object "/home/vvdn/rejohn/openSRS_newbr/srsRAN_Project/build/apps/gnb/gnb", at 0x55d50d33d74e, in std::thread::_State_impl<std::thread::_Invoker<std::tuple<srsran::unique_thread::make_thread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, srsran::unique_function<void (), 32ul, false>, srsran::os_thread_realtime_priority, srsran::os_sched_affinity_bitmask const&)::{lambda()#1}> > >::_M_run() #12 Object "/home/vvdn/rejohn/openSRS_newbr/srsRAN_Project/build/apps/gnb/gnb", at 0x55d50d3c1632, in std::_Function_handler<void (), srsran::priority_task_worker_pool::create_pop_loop_task()::{lambda()#1}>::_M_invoke(std::_Any_data const&) #11 Object "/home/vvdn/rejohn/openSRS_newbr/srsRAN_Project/build/apps/gnb/gnb", at 0x55d50d35fd8e, in srsran::detail::task_strand_impl<srsran::priority_task_worker_pool_executor, srsran::detail::priority_strand_queue>::run_enqueued_tasks() #10 Object "/home/vvdn/rejohn/openSRS_newbr/srsRAN_Project/build/apps/gnb/gnb", at 0x55d50d4120e1, in srsran::task_details::heap_table_t<srsran::srs_cu_up::f1u_bearer_impl::handle_pdu(srsran::nru_ul_message)::{lambda()#1}, void>::call(void*) const #9 Object "/home/vvdn/rejohn/openSRS_newbr/srsRAN_Project/build/apps/gnb/gnb", at 0x55d50d41153d, in srsran::srs_cu_up::f1u_bearer_impl::handle_pdu_impl(srsran::nru_ul_message) #8 Object "/home/vvdn/rejohn/openSRS_newbr/srsRAN_Project/build/apps/gnb/gnb", at 0x55d50cf4245e, in srsran::srs_cu_up::f1u_pdcp_adapter::on_new_sdu(srsran::byte_buffer_chain) #7 Object "/home/vvdn/rejohn/openSRS_newbr/srsRAN_Project/build/apps/gnb/gnb", at 0x55d50d6214d1, in srsran::pdcp_entity_rx::handle_pdu(srsran::byte_buffer_chain) #6 Object "/home/vvdn/rejohn/openSRS_newbr/srsRAN_Project/build/apps/gnb/gnb", at 0x55d50d620d62, in srsran::pdcp_entity_rx::handle_data_pdu(srsran::byte_buffer) #5 Object "/home/vvdn/rejohn/openSRS_newbr/srsRAN_Project/build/apps/gnb/gnb", at 0x55d50d61ec96, in srsran::pdcp_entity_rx::deliver_all_consecutive_counts() #4 Object "/home/vvdn/rejohn/openSRS_newbr/srsRAN_Project/build/apps/gnb/gnb", at 0x55d50cf401ce, in srsran::srs_cu_up::pdcp_sdap_adapter::on_new_sdu(srsran::byte_buffer) #3 Object "/home/vvdn/rejohn/openSRS_newbr/srsRAN_Project/build/apps/gnb/gnb", at 0x55d50d41a833, in srsran::srs_cu_up::sdap_entity_rx_impl::handle_pdu(srsran::byte_buffer) #2 Object "/home/vvdn/rejohn/openSRS_newbr/srsRAN_Project/build/apps/gnb/gnb", at 0x55d50cf4043e, in srsran::srs_cu_up::sdap_gtpu_adapter::on_new_sdu(srsran::byte_buffer, srsran::qos_flow_id_t) #1 Object "/home/vvdn/rejohn/openSRS_newbr/srsRAN_Project/build/apps/gnb/gnb", at 0x55d50d683f18, in srsran::gtpu_tunnel_ngu_tx_impl::handle_sdu(srsran::byte_buffer, srsran::qos_flow_id_t) #0 Object "/home/vvdn/rejohn/openSRS_newbr/srsRAN_Project/build/apps/gnb/gnb", at 0x55d50cf285c8, in srsran::srs_cu_up::gtpu_network_gateway_adapter::on_new_pdu(srsran::byte_buffer, sockaddr_storage const&) Segmentation fault (Invalid permissions for mapped object [0x55d510dea]) Segmentation fault (core dumped)
##Setup Details##
The block diagram is shown below:
Here we have tried with only one RU by sending a throughput of 1.2 gbps
gNB trace:
Also you can find that 1.2 gbps throughput is not achieved here.
The configuration file for the gNB is given below:
gnb_ru_ran_vvdn_lpru_tdd_n78_100mhz_4x4.zip
Thanks @vvdn5gtest, is the core open5gs here?
Hi @vvdn5gtest, two questions:
- Did the crash happen on shutdown, because the output starts with "Stopping .. "?
- Moreover it seems a UL PDU caused the crash - was the UE transmitting in UL direction when the crash happened?
Hi @vvdn5gtest, one more follow up question, how reproducible is this? Does it happen every time or only sporadically?
@vvdn5gtest, I was not able to reproduce yet, but we spotted some issues in the termination of the CU-UP.
Could you try the following patch and let me know if it helps? Just do git am < name_of_the_patch