TestNG testng-failed.xml 'invocation-numbers' values are not calculated correctly with retry and dataproviders

[gyizol]

TestNG Version

7.10.2

Expected behavior

testng-failed.xml contains correct 'invocation-numbers' indexes for methods with data-provider and retries.

Actual behavior

testng-failed.xml contains incorrect 'invocation-numbers' indexes for methods with data-provider and retries.

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

RetryAnalyzer.java

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryAnalyzer implements IRetryAnalyzer {

    int counter = 0;
    int retryLimit = 1;

    /*
     * Retry method
     */
    public boolean retry(ITestResult result) {
        if (!result.isSuccess()) {
            if (counter < retryLimit) {
                System.out.println(" ... FAILURE! Retrying ... (#" + (counter + 1) + " out of " + retryLimit + ")");
                counter++;
                return true;
            }
        }
        return false;
    }
}

RetryTest.java

import static org.testng.Assert.assertEquals;
import java.util.Random;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class RetryTest {

    @Test(retryAnalyzer = RetryAnalyzer.class, dataProvider = "dp")
    public void testDataProvider(int number) {
        Random random = new Random();
        int randomNumber = random.nextInt(3) + 1;  // Generates a number between 1 and 3
        System.out.println("Number from data-provider: " + number);
        System.out.println("Generated number: " + randomNumber);
        assertEquals(randomNumber, number, "The two values are different!");
    }

    @DataProvider(name = "dp")
    public Object[][] createTestData() {
        return new Object[][] { { 1 }, { 2 }, { 3 } };
    }
}

Example output of a test run:

Number from data-provider: 1
Generated number: 2
 ... FAILURE! Retrying ... (#1 out of 1)
Number from data-provider: 1
Generated number: 1
Number from data-provider: 2
Generated number: 1
 ... FAILURE! Retrying ... (#1 out of 1)
Number from data-provider: 2
Generated number: 1
Number from data-provider: 3
Generated number: 1
 ... FAILURE! Retrying ... (#1 out of 1)
Number from data-provider: 3
Generated number: 3
PASSED: restAPITest.boards.BoardsTest.testDataProvider(3)
PASSED: restAPITest.boards.BoardsTest.testDataProvider(1)
FAILED: restAPITest.boards.BoardsTest.testDataProvider(2)

testng-failed.xml generated from the test run:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Failed suite [Default suite]" guice-stage="DEVELOPMENT">
  <test thread-count="5" name="Default test(failed)">
    <classes>
      <class name="RetryTest">
        <methods>
          <include name="testDataProvider" invocation-numbers="0 1 2"/>
        </methods>
      </class> <!-- RetryTest -->
    </classes>
  </test> <!-- Default test(failed) -->
</suite> <!-- Failed suite [Default suite] -->

Notice the invocation-numbers part which includes all the indexes (i.e.: the 0th, 1st, and 2nd value of the data-provider) to be executed for a rerun. I would expect that invocation-numbers would contain only those indexes that failed completely - meaning after all retries. (In this case only the index value '1' should be there, since that represents the value '2' of the data-provider.)