eficode/robotframework-oxygen

CICD is not failing when listener is failing

Tattoo opened this issue · 0 comments

The CICD currently relies on Robot Framework exit code to know if the test execution failed or not. However, in cases where the Oxygen listener fails, but the test execution otherwise passes, RF will return 0 and the CICD passes.

For example:

==============================================================================
Atest                                                                 | PASS |
10 critical tests, 10 passed, 0 failed
10 tests total, 10 passed, 0 failed
==============================================================================
[ ERROR ] Calling method 'output_file' of listener 'oxygen.listener' failed: MismatchArgumentException: parse_results expects 3 arguments but got 1
Output:  /[...]/output.xml
Log:     /[...]/log.html
Report:  /[...]/report.html

[python38][robotframework==3.2] Exited with 0

Since Oxygen is, in part, a listener, we need CICD to detect that listener has failed during the test execution