Snooz82/robotframework-datadriver

DataDriver.rerunfailed executes Passed tests if there are no failures in the original test

Closed this issue · 4 comments

eldaduzman commented

When re-executing a test suite with a --prerunmodifier DataDriver.rerunfailed I see that it executes again all the tests if there are no failures in the original execution.

look at the attached 7zip file

When executing run_has_failure.bat I see that only the failed test is executed in the second and third attempt.
However, when executing run_all_passes.bat, the three passed tests are executed again in the two reruns.

This causes massive overheads in the execution of test pipelines.

I tried it with windows 10 pro and python version 3.7.4.
Versions of robotframework and robotframework-datadriver are in the requirements.txt file in the zip.

Note, I also tried it with robotframework-datadriver version 1.0.0 and got similar results.

eldaduzman commented

Hi @Snooz82 , thanks for the quick fix.

datadriver now only re-runs failed tests and doesn't re-run passed tests in case of all passes.

But now the problem is that the re-run process ends with an error:

[ ERROR ] Suite 'Tests All Passes' contains no tests after model modifiers

exit code if 252.

I think this is a mistake because if there's nothing to re-run the process should not return an error.
This creates a lot of extra burden in the testing pipelines of error checking after every re-try.

WDYT?

Snooz82 commented

Hi @eldaduzman

i thought so as well, but the error is also there with the same code, when you use Robots own option --rerunfailed

So Robot brings a different Error but also Error code 252

i think DataDriver should behave same as Robot Framework.

  • 👍1
eldaduzman commented

I see.
I agree, the behavior should be consistent.

Is there a "best practice" when re-runing tests?