reportportal/agent-java-karate

Rerun failed scenarios

Closed this issue · 1 comments

Snagsvkrv commented

Need guidance or ways
In order to rerun flaky tests in karate, we are using identifier as @flaky to determine which tests needs to be retried.

List<ScenarioResult> flakyResults = new ArrayList<ScenarioResult>();
        flakyResults = results.getScenarioResults()
                .filter(scenarioResult -> {
                    List<Tag> tags = scenarioResult.getScenario().getTags();
                    boolean hasFlakyTag = tags.stream()
                            .map(Tag::getName)
                            .anyMatch(tagName -> tagName.trim().equalsIgnoreCase("flaky"));
                    return hasFlakyTag && scenarioResult.isFailed();
                })
                .collect(Collectors.toList());

Results are being fetched using

results = Runner.path("classpath:abc")
                    .outputCucumberJson(true)
                    .hook(new ReportPortalHook())
                    .outputJunitXml(true)
                    .tags("~@ignore")
                    .parallel(threads);

Retrial for tests is being achieved via:

results.getSuite().retryScenario(scenarioResult.getScenario());

This causes below exception:

13:39:36.825 [] [pool-6-thread-3] ERROR c.epam.reportportal.service.Launch - ItemID should not be null
java.lang.NullPointerException: ItemID should not be null
	at com.epam.reportportal.service.LaunchImpl.finishTestItem(LaunchImpl.java:390)
	at com.epam.reportportal.karate.ReportPortalHook.afterScenario(ReportPortalHook.java:297)
	at com.intuit.karate.core.ScenarioRuntime.lambda$afterRun$3(ScenarioRuntime.java:507)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at com.intuit.karate.core.ScenarioRuntime.afterRun(ScenarioRuntime.java:507)
	at com.intuit.karate.core.ScenarioRuntime.run(ScenarioRuntime.java:412)
	at com.intuit.karate.Suite.retryScenario(Suite.java:330)
	at BaseTest.retryScenarioWithRetries(BaseTest.java:151)
	at BaseTest.lambda$retryFlakyTestsParallel$2(BaseTest.java:122)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

and on UI for each step new child elements gets created:
Screenshot 2024-09-18 at 1 41 15 PM

Additionaly
We cannot update reportportal properties file to add launch_id and rerun true property to it when executing it via pipeline and even though if we re-create properties file with rerun data launching a new Runner won't update failed tests with retry tags but create new instances in same report.
Screenshot 2024-09-18 at 1 39 43 PM

How to reproduce
say sample.feature:

@testUi @flaky
  Scenario Outline: Demo UI Test flaky sample
    * karate.log('Running for username <userName>')
    * match 1 == 2

    Examples:
      | userName               | Password     |
      | egyptuat_2@somemail.com | Welcome1@123 |
      | egyptuat_2@somemail.com | Welcome1@123 |
      | egyptuat_2@somemail.com | Welcome1@123 |

  @testUi @flaky
  Scenario: responseCheck
    * def responseCheck =
      """
      function getCurrentMinute() {
        var now = new Date();
        var seconds = now.getSeconds();
        if (seconds > 45)
        return seconds;
        else
        throw new Error("Seconds is less than 45");
      }
      """

    * def seconds = retry(5,5000,responseCheck)
    * print seconds

using
https://mvnrepository.com/artifact/com.epam.reportportal/agent-java-karate/5.0.5

@HardNorth any guidance on this would be helpful, even if you can point me into correct direction also will be helpful.