spring-projects/spring-aot-smoke-tests

Avoid GitHub sending build failure notifications for smoke tests that are not expected to pass

wilkinsona opened this issue · 1 comments

We have a number of smoke tests that are not expected to pass. Right now, each daily run of these smoke tests fails and causes GitHub to send a notification:

Notifications for scheduled workflows are sent to the user who last modified the cron syntax in the workflow file

These notifications are unwanted noise as they aren't really telling us anything. It would be useful if we could configure, on a per-test basis, that the expected outcome is failure. Something like this in an individual test's build.gradle:

aotSmokeTest {
    expectedOutcomes {
        nativeAppTest = failure
        nativeTest = failure
    }
}

This would translate to setting continue-on-error in the relevant step of smoke-test-jvm.yml and/or smoke-test-native.yml using an input from the calling workflow.

If we wish, we could then go a step further and fail the build if a test passes unexpectedly by adding a subsequent step that fails if the step that ran the test passed unexpectedly.

This would translate to setting continue-on-error in the relevant step of smoke-test-jvm.yml and/or smoke-test-native.yml using an input from the calling workflow.

Done in 537c409.

If we wish, we could then go a step further and fail the build if a test passes unexpectedly by adding a subsequent step that fails if the step that ran the test passed unexpectedly.

Done in e36fa98.