Avoid GitHub sending build failure notifications for smoke tests that are not expected to pass
wilkinsona opened this issue · 1 comments
We have a number of smoke tests that are not expected to pass. Right now, each daily run of these smoke tests fails and causes GitHub to send a notification:
Notifications for scheduled workflows are sent to the user who last modified the cron syntax in the workflow file
These notifications are unwanted noise as they aren't really telling us anything. It would be useful if we could configure, on a per-test basis, that the expected outcome is failure. Something like this in an individual test's build.gradle
:
aotSmokeTest {
expectedOutcomes {
nativeAppTest = failure
nativeTest = failure
}
}
This would translate to setting continue-on-error
in the relevant step of smoke-test-jvm.yml
and/or smoke-test-native.yml
using an input from the calling workflow.
If we wish, we could then go a step further and fail the build if a test passes unexpectedly by adding a subsequent step that fails if the step that ran the test passed unexpectedly.
This would translate to setting continue-on-error in the relevant step of smoke-test-jvm.yml and/or smoke-test-native.yml using an input from the calling workflow.
Done in 537c409.
If we wish, we could then go a step further and fail the build if a test passes unexpectedly by adding a subsequent step that fails if the step that ran the test passed unexpectedly.
Done in e36fa98.