Investigate flaky multithreading tests
ulupo opened this issue · 2 comments
The tests for consistency of multithreaded results with single-threaded results are flaky, see e.g. https://github.com/giotto-ai/giotto-ph/runs/7875382240?check_suite_focus=true.
I'm not saying this implies there is a bug in the C++ backend, but it would be good to know exactly why they sometimes fail.
I might be wrong, but looking at the logs, the issue seems related to too slow:
Unreliable test timings! On an initial run, this test took 640.08ms, which exceeded the deadline of 500.00ms, but on a subsequent run it took 62.90 ms, which did not. If you expect this sort of variability in your test timings, consider turning deadlines off for this test by setting deadline=None.
I will investigate deeper there, but it might be less critical that I first ready your description 😅
There are definitely assertion errors in some of these tests too, but I linked to the wrong one!