Gradle retry plugin: Unable to handle failed test that has a random unique name per iteration
Opened this issue · 1 comments
The issue occurs when the parameterized Spock method with a random unique name for each iteration fails.
As a result, the Gradle retry plugin can not retrieve old failed test details by new rerun test details.
Issue:
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':functional-tests:functionalTest'.
> The following test methods could not be retried, which is unexpected. Please file a bug report at https://github.com/gradle/test-retry-gradle-plugin/issues
As a solution, tests that have such kind of naming strategy should be modified.
Example:
...EnvExtension:-2 - Creating topologies (1)
EnvExtension:-1 - Starting topology-related HC: lab_id=3
EnvExtension:-1 - Topology-related HC passed: lab_id=3
...
BaseSpecification:-2 - Booked lab with id 3 for spec HaFlowStatSpec, thread: Thread[Test worker,5,main]. sw: 00:00:00:00:00:00:00:01
...
INFO LogParallelSpecsListener:-2 - Running test: HaFlowStatSpec['LOW_PRIORITY', 'HA_FLOW']#System is able to collect latency stats for subflows [subFlow: 11Apr091705_157_oatflour2420_haflow-a, direction: FORWARD, #0]
ERROR LogParallelSpecsListener:-2 - Specs ran in parallel: [HaFlowStatSpec]
HaFlowStatSpec > System is able to collect latency stats for subflows ✘ System is able to collect latency stats for subflows [subFlow: 11Apr091705_157_oatflour2420_haflow-a, direction: FORWARD, #0] (1m 1s)
Condition failed with Exception:
wait(statsRouterRequestInterval) { if(direction == FORWARD && subFlow.contains("haflow-a")) { assert labService.getLabs().first().labId != 3 } assert flowStats.of(subFlow).get(FLOW_RTT, direction).hasNonZeroValues() }
at org.openkilda.functionaltests.spec.stats.HaFlowStatSpec.System is able to collect latency stats for subflows(HaFlowStatSpec.groovy:100)
Caused by: org.openkilda.functionaltests.helpers.Wrappers$WaitTimeoutException: Condition was not satisfied within 60.0 seconds
at org.openkilda.functionaltests.helpers.Wrappers.wait(Wrappers.groovy:109)
... 1 more
Caused by: Condition not satisfied:
labService.getLabs().first().labId != 3 // intentional failure to trigger the 1st failure and during rerun the new topology is created and test should pass
Retry by Gradle due to the failure:
HaFlowStatSpec STANDARD_OUT
EnvExtension:-1 - Starting topology-related HC: lab_id=4
...
BaseSpecification:-2 - Booked lab with id 4 for spec HaFlowStatSpec, thread: Thread[Test worker,5,main]. sw: 00:00:00:00:00:00:00:01
HaFlowStatSpec > System is able to collect latency stats for subflows > org.openkilda.functionaltests.spec.stats.HaFlowStatSpec.System is able to collect latency stats for subflows [subFlow: 11Apr091946_456_redlentils4861_haflow-b, direction: FORWARD, #1] STANDARD_OUT
2024-04-11T09:20:22,280 INFO LogParallelSpecsListener:-2 - Running test: HaFlowStatSpec['LOW_PRIORITY', 'HA_FLOW']#System is able to collect latency stats for subflows [subFlow: 11Apr091946_456_redlentils4861_haflow-b, direction: FORWARD, #1]
HaFlowStatSpec > System is able to collect latency stats for subflows ✔ System is able to collect latency stats for subflows [subFlow: 11Apr091946_456_redlentils4861_haflow-b, direction: FORWARD, #1] - //test passed after retry
> Task :functional-tests:functionalTest FAILED
:functional-tests:functionalTest (Thread[Execution worker for ':' Thread 4,5,main]) completed. Took 4 mins 20.883 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':functional-tests:functionalTest'.
> The following test methods could not be retried, which is unexpected. Please file a bug report at https://github.com/gradle/test-retry-gradle-plugin/issues
org.openkilda.functionaltests.spec.stats.HaFlowStatSpec#System is able to collect latency stats for subflows [subFlow: 11Apr091705_157_oatflour2420_haflow-a, direction: FORWARD, #0]
Failed test:
System is able to collect latency stats for subflows [subFlow: 11Apr091705_157_oatflour2420_haflow-a, direction: FORWARD, #0]
Test name after Gradle rerun failed test^:
System is able to collect latency stats for subflows [subFlow: 11Apr091946_456_redlentils4861_haflow-b, direction: FORWARD, #1]
The following method naming also caused the same Gradle issue with report generation in the scope of the retry mechanism.
@Unroll("Able to create a second flow if #data.description")
def "Able to create multiple flows on certain combinations of switch-port-vlans"() {
The appropriate changes have been added in the scope #5644