FPGA Python interpreter segfaults

Question

FPGA Python interpreter segfaults

Opened this issue 5 months ago · 0 comments

Three FPGA tests cause segfaults in the Python interpreter. (This is based on running FPGA CI with commit 062405a which disables these tests which results in no more Python segfaults)

The affected tests are:

tests/blas/nodes/axpy_test.py::test_axpy_fpga_array
tests/blas/nodes/axpy_test.py::test_axpy_fpga_stream
tests/fpga/veclen_conversion_connector_test.py::test_veclen_conversion_connector

In the FPGA CI log, they result in the following stack traces:

Fatal Python error: Segmentation fault

Current thread 0x00007fabf0896740 (most recent call first):
  File "/home/timos/actions-runner/_work/dace/dace/dace/codegen/compiled_sdfg.py", line 435 in fast_call
  File "/home/timos/actions-runner/_work/dace/dace/dace/codegen/compiled_sdfg.py", line 404 in __call__
  File "/home/timos/actions-runner/_work/dace/dace/tests/blas/nodes/axpy_test.py", line 48 in run_test
  File "/home/timos/actions-runner/_work/dace/dace/tests/blas/nodes/axpy_test.py", line 118 in test_axpy_fpga_array
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 115 in _run_fpga_test
  File "/usr/lib/python3.11/multiprocessing/process.py", line 108 in run
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 70 in run
  File "/usr/lib/python3.11/multiprocessing/process.py", line 314 in _bootstrap
  File "/home/timos/.venv/lib/python3.11/site-packages/coverage/multiproc.py", line 50 in _bootstrap
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 71 in _launch
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 19 in __init__
  File "/usr/lib/python3.11/multiprocessing/context.py", line 281 in _Popen
  File "/usr/lib/python3.11/multiprocessing/context.py", line 224 in _Popen
  File "/usr/lib/python3.11/multiprocessing/process.py", line 121 in start
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 217 in internal
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 233 in wrapper
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/python.py", line 1792 in runtest
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 174 in run_one_test
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 157 in pytest_runtestloop
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 355 in <module>
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1157 in executetask
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 296 in run
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 361 in _perform_spawn
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 343 in integrate_as_primary_thread
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1142 in serve
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1640 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>

Current thread 0x00007fabf0896740 (most recent call first):
  File "/home/timos/actions-runner/_work/dace/dace/dace/codegen/compiled_sdfg.py", line 435 in fast_call
  File "/home/timos/actions-runner/_work/dace/dace/dace/codegen/compiled_sdfg.py", line 404 in __call__
  File "/home/timos/actions-runner/_work/dace/dace/tests/blas/nodes/axpy_test.py", line 48 in run_test
  File "/home/timos/actions-runner/_work/dace/dace/tests/blas/nodes/axpy_test.py", line 124 in test_axpy_fpga_stream
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 115 in _run_fpga_test
  File "/usr/lib/python3.11/multiprocessing/process.py", line 108 in run
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 70 in run
  File "/usr/lib/python3.11/multiprocessing/process.py", line 314 in _bootstrap
  File "/home/timos/.venv/lib/python3.11/site-packages/coverage/multiproc.py", line 50 in _bootstrap
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 71 in _launch
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 19 in __init__
  File "/usr/lib/python3.11/multiprocessing/context.py", line 281 in _Popen
  File "/usr/lib/python3.11/multiprocessing/context.py", line 224 in _Popen
  File "/usr/lib/python3.11/multiprocessing/process.py", line 121 in start
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 217 in internal
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 233 in wrapper
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/python.py", line 1792 in runtest
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 174 in run_one_test
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 157 in pytest_runtestloop
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 355 in <module>
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1157 in executetask
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 296 in run
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 361 in _perform_spawn
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 343 in integrate_as_primary_thread
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1142 in serve
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1640 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>

Fatal Python error: Segmentation fault

Current thread 0x00007fab4ae49740 (most recent call first):
  File "/home/timos/actions-runner/_work/dace/dace/dace/codegen/compiled_sdfg.py", line 435 in fast_call
  File "/home/timos/actions-runner/_work/dace/dace/dace/codegen/compiled_sdfg.py", line 404 in __call__
  File "/home/timos/actions-runner/_work/dace/dace/dace/sdfg/sdfg.py", line 2334 in __call__
  File "/home/timos/actions-runner/_work/dace/dace/tests/fpga/veclen_conversion_connector_test.py", line 22 in test_veclen_conversion_connector
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 115 in _run_fpga_test
  File "/usr/lib/python3.11/multiprocessing/process.py", line 108 in run
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 70 in run
  File "/usr/lib/python3.11/multiprocessing/process.py", line 314 in _bootstrap
  File "/home/timos/.venv/lib/python3.11/site-packages/coverage/multiproc.py", line 50 in _bootstrap
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 71 in _launch
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 19 in __init__
  File "/usr/lib/python3.11/multiprocessing/context.py", line 281 in _Popen
  File "/usr/lib/python3.11/multiprocessing/context.py", line 224 in _Popen
  File "/usr/lib/python3.11/multiprocessing/process.py", line 121 in start
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 217 in internal
  File "/home/timos/actions-runner/_work/dace/dace/dace/fpga_testing.py", line 233 in wrapper
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/python.py", line 1792 in runtest
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 174 in run_one_test
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 157 in pytest_runtestloop
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/home/timos/.venv/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/timos/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/timos/.venv/lib/python3.11/site-packages/xdist/remote.py", line 355 in <module>
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1157 in executetask
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 296 in run
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 361 in _perform_spawn
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 343 in integrate_as_primary_thread
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1142 in serve
  File "/home/timos/.venv/lib/python3.11/site-packages/execnet/gateway_base.py", line 1640 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>