serenity-bdd/serenity-cucumber-starter

Flaky tests, because exceptions in tests can be ignored

Opened this issue · 0 comments

When running tests with cucumber/serenity runner exceptions can be ignored, resulting in runs which pass but that should have failed. To reproduce there is a repo fork with branch with a test to reproduce the issue.

Exceptions in tests do get ignored when:

  • @Step annotation is used in single scenario
  • or atleast one scenario uses @Step annotations in feature file run

Attaching a picture from 2 runs:

  • both scenarios DOES throw exception
  • when ran as a single test without @Step annotation -- test fails (picture on the left)
  • when ran as a whole feature -- tests pass even though both tests should fail (picture on the right)

image