Unexpected behaviour when running regression test
kursatyurt opened this issue · 3 comments
Describe the bug
During the regression tests of #1977, I noticed that some runs weirdly does not pass the test. After the third attempt regression test passed successfully.
Please see the attempts https://github.com/su2code/SU2/actions/runs/4480654814/jobs/7883573846 hybrid_regression_AD.py.
Looks weird to me.
Maybe a little bit of non determinism because of atomic adjoint updates. @jblueh by chance did you run thread sanitizer after the codi 2 update?
Could be a data race, thanks for reporting @kursatyurt. Last I ran the thread sanitizer during the CoDiPack 2 update, nothing came up at the time. The failing test is the recently added pywrapper_CFD_AD_MeshDisp. I'll look into it.
I reran the thread sanitizer analysis and it didn't detect any data races. The deviations are small, so I'm with @pcarruscag that this is probably inherent non-determinism. I relaxed the tolerances in #1984.