DARIAEngineering/dcaf_case_management

flaky tests

Opened this issue · 2 comments

xmunoz commented

Thanks for creating an issue! Please fill out this form so we can be sure to have all the information we need, and to minimize back and forth.

  • What are we trying to do?
    Eliminate test flakiness

  • What feature or behavior is this required for?
    Quickly and easily reviewing PRs, and merging with confidence

  • How could we solve this issue? (Not knowing is okay!)
    I don't know

  • Anything else?

Here are the flaky tests:

  • UpdatePatientInfoTest::changing fulfillment information#test_0001_should alter the information (OptionalRetry set up)
  • LoggingCallsTest::logging Left voicemail#test_0002_should be visible on the call log after clicking Left voicemail (OptionalRetry set up)
  • LoggingCallsTest::logging Left voicemail#test_0001_should close the modal when clicking Left voicemail (OptionalRetry set up)
  • LoggingCallsTest::logging reached patient#test_0002_should be viewable on the call log (OptionalRetry set up)
  • LoggingCallsTest::logging reached patient#test_0001_should redirect to the edit view when a patient has been reached: (OptionalRetry set up)
  • LoggingCallsTest::verifying modal behavior and content#test_0001_should open a modal when clicking the call glyphicon OptionalRetry set up)

These are actually errors, not failures. Example: https://github.com/DARIAEngineering/dcaf_case_management/actions/runs/6327925858/job/17184902734

Sometimes we also get this error:

There will either be 6 errors (above) or 1 error (below) for a given test run. I've never seen a case where there are 7 errors.

colinxfleming commented

I was hoping this was going to be low hanging fruit that we could slap extend Minitest::OptionalRetry (which will retry a failing test up to three times before declaring defeat) but this sounds like something's up with those, LoggingCallsTest in particular.

xmunoz commented

I was going to say that we haven't see this in a while, but looking at recent test runs, here is a recent example of a flaky failure (second failure mode): https://github.com/DARIAEngineering/dcaf_case_management/actions/runs/6785077889/job/18442620959