When there are failed tests test execution process as a whole is marked as failed with status 1
PavelSkopik opened this issue · 1 comments
When there is at least one failed test the tests execution process always ends as a failure with exit code 1.
When running tests on a CI server - e.g. Jenkins - tests failures mark the job as failed because the process return code is 1 even though the run was successful.
Would it be possible to mark failed tests process as a successful process or provide a CLI option to control that state? That way the success or job stability can be determined based on test results gathered by CI server after the tests are done.
There would also be a distinction between actual failure such as the test URL not being available or the test execution failing for other reason.
Thank your for considering this.
Pavel
Closing as part of the triage, since the new test-runner is quite different. Please reopen if its still persistent.
For your specific request, there are multiple ways of fixing that. npm run test || true <- never fail on the CI, since the command always returns exit code 0 would probably the easiest. Also you can annotate your tests using test.fail()or test.fixme() see here: https://playwright.dev/docs/test-annotations