LLNL/serac

Nondeterministic test failure

btalamini opened this issue · 1 comments

During a PR, I got an unexpected failure in a test in the file dynamic_solid_adjoint.cpp. I say "unexpected", becaues the code introduced in my PR isn't used in that test. The failure looks like this:

48: /usr/WS1/bramwell/gl/builds/xYGwyxK_/000/gitlab/smith/serac/src/serac/physics/tests/dynamic_solid_adjoint.cpp:295: Failure
48: The difference between directional_deriv and (qoi_plus - qoi_base) / eps is nan, which exceeds eps, where
48: directional_deriv evaluates to -nan,
48: (qoi_plus - qoi_base) / eps evaluates to -0.023019492978448852, and
48: eps evaluates to 1.9999999999999999e-07.
48: 
48: [  FAILED  ] SolidMechanicsSensitivityFixture.ShapeSensitivities (7280 ms)

I'm suspicious that we have some kind of non-deterministic problem here, like a race condition. The entire log is included below.

output.log.make.test.txt

There was an indexing error (going past the end of the array). The error was intermittent, because the retrieved value was being multiplied by 0 - it wouldn't affect the results, so long as the memory read beyond the end of the array could be interpreted as a double. Otherwise, a NAN was issued and would cause the test to fail. This is closed by #1183