Motr panic at the end of Maestro's `check_pm_interlock` test that does one transport through mio.
chhaine opened this issue · 3 comments
This is the log for check_pm_interlock. m0trace file is too big (67M) for github, please specify a transfer method if you need it.
1 successful of 1 run
SUCCESS
motr[28947]: 5c90 ERROR [rpc/item.c:1683:m0_rpc_item_xid_list_fini] session=0x220a558 item=0x7f57580cc898 [REQUEST/M0_IOSERVICE_COB_CREATE_OPCODE(45)] ri_sm.sm_state=7
motr[28947]: 5c90 ERROR [rpc/item.c:1683:m0_rpc_item_xid_list_fini] session=0x220a558 item=0x7f57580c8898 [REQUEST/M0_IOSERVICE_COB_CREATE_OPCODE(45)] ri_sm.sm_state=7
motr[28947]: 5b50 FATAL [lib/assert.c:50:m0_panic] panic: (m0_list_is_empty(head)) at m0_list_fini() (lib/list.c:38) [git: sage-base-1.0-187-g4b3f850] /home/vagrant/maestro-core/tests/m0trace.28947
Motr panic: (m0_list_is_empty(head)) at m0_list_fini() lib/list.c:38 (errno: 11) (last failed: none) [git: sage-base-1.0-187-g4b3f850] pid: 28947 /home/vagrant/maestro-core/tests/m0trace.28947
/lib64/libmotr.so.1(m0_arch_backtrace+0x2f)[0x7f57c4bd52ff]
/lib64/libmotr.so.1(m0_arch_panic+0xf3)[0x7f57c4bd54e3]
/lib64/libmotr.so.1(+0x339a04)[0x7f57c4bc5a04]
/lib64/libmotr.so.1(+0x33cecb)[0x7f57c4bc8ecb]
/lib64/libmotr.so.1(m0_rpc_item_xid_list_fini+0xde)[0x7f57c4c51c7e]
/lib64/libmotr.so.1(m0_rpc_session_fini_locked+0x7d)[0x7f57c4c5cb6d]
/lib64/libmotr.so.1(m0_rpc_session_fini+0x65)[0x7f57c4c5cc95]
/lib64/libmotr.so.1(m0_rpc_link_fini+0x47)[0x7f57c4c55997]
/lib64/libmotr.so.1(m0_reqh_service_ctx_fini+0xc5)[0x7f57c4c37745]
/lib64/libmotr.so.1(m0_reqh_service_ctx_destroy+0x11)[0x7f57c4c37b41]
/lib64/libmotr.so.1(+0x39e588)[0x7f57c4c2a588]
/lib64/libmotr.so.1(m0_pools_service_ctx_destroy+0x61)[0x7f57c4c2cee1]
/lib64/libmotr.so.1(+0x3629a3)[0x7f57c4bee9a3]
/lib64/libmotr.so.1(+0x3d554c)[0x7f57c4c6154c]
/lib64/libmotr.so.1(m0_client_fini+0x158)[0x7f57c4bf0af8]
/home/vagrant/maestro-core/.libs/libmaestro.so.0(+0x18d825)[0x7f57c53b3825]
/home/vagrant/maestro-core/.libs/libmaestro.so.0(mio_fini+0x3a)[0x7f57c53b181a]
/lib64/libc.so.6(+0x39ce9)[0x7f57c33a1ce9]
/lib64/libc.so.6(+0x39d37)[0x7f57c33a1d37]
/lib64/libc.so.6(__libc_start_main+0xfc)[0x7f57c338a55c]
../tests/simple_interlock_client_2[0x4042f6]
Could you give me the following details? (1) The version of Maestro; (2) How you run the Maestro tests; (3) your Motr version; (4) do you run your test on vm or a cluster? (5) Does the panic happen very time or occasionally when running the test?
you can upload the file on Maestro slack, and we can discuss further there
(1) Please use the following branch:
https://gitlab.jsc.fz-juelich.de/maestro/maestro-core/-/tree/17-job-failed-282354-needs-update-of-mio-interface-and-build-rules
(MAESTRO 0.1.741-73b9-dirty)
(2) both make check and ./check_pm_interlock.sh directly produce the error every time ((5)).
(3)
$ rpm -qa | grep motr-
cortx-motr-devel-1.0.0-1_git4b3f850_3.10.0_1062.el7.x86_64
cortx-motr-1.0.0-1_git4b3f850_3.10.0_1062.el7.x86_64
(4) I run with the Maestro VM:
https://gitlab.jsc.fz-juelich.de/maestro/maestro-mero-vm
(5) see (2)
Solved, issue was not calling mio_op_fini() after mio_cmd_wait_on_op().