Test cases are being skipped when running in parallel a factory class containing dependent methods

Question

Test cases are being skipped when running in parallel a factory class containing dependent methods

galinaff opened this issue 9 months ago · 11 comments

galinaff commented 9 months ago

TestNG Version

7.0.0

Expected behavior

All the test case cases should be executed.

Actual behavior

Only the first test case is executed by both threads and then another test class setup is done.

Is the issue reproducible on runner?

Note:
In our case, it is important to keep setGroupByInstances(true) and setPreserveOrder(false)

Test case sample

package tests.testCases;

import java.util.ArrayList;

import org.testng.TestNG;
import org.testng.xml.XmlClass;
import org.testng.xml.XmlSuite;
import org.testng.xml.XmlTest;

public class main
{
    public static void main(String[] args) throws ClassNotFoundException
    {
        XmlSuite suite;
        XmlTest test;
        TestNG testNG;
        XmlClass xmlClass;
        ArrayList<XmlTest> tests;
        ArrayList<XmlClass> testClasses;
        ArrayList<XmlSuite> testSuites;

        testNG = new TestNG();
        testNG.setVerbose(0);

        testSuites = new ArrayList<XmlSuite>();
        suite = new XmlSuite();
        suite.setName("Suite");

        test = new XmlTest();
        test.setName("testCases");
        test.setSuite(suite);
        test.setGroupByInstances(true);
        test.setPreserveOrder(false);

        testClasses = new ArrayList<XmlClass>();
        xmlClass = new XmlClass(Class.forName("tests.testCases.TestCaseB"));
        testClasses.add(xmlClass);
        test.setClasses(testClasses);

        tests = new ArrayList<XmlTest>();
        tests.add(test);

        suite.setTests(tests);
        suite.setGroupByInstances(true);
        suite.setPreserveOrder(false);
        suite.setParallel(XmlSuite.ParallelMode.INSTANCES);
        suite.setThreadCount(2);

        testSuites.add(suite);

        testNG.setXmlSuites(testSuites);

        testNG.run();
    }
}

Test case sample

package tests.testCases;

import org.testng.ITestContext;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;

public class TestCaseA
{
    @BeforeClass (alwaysRun=true)
    protected void setUpTestClass(ITestContext context)
    {
        System.out.println("setUpTestClass with thread id " + Thread.currentThread().getId());
    }

    @AfterClass (alwaysRun=true)
    protected void tearDownTestClass()
    {
        System.out.println("tearDownTestClass with thread id " + Thread.currentThread().getId());
    }
}

Test case sample

package tests.testCases;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;

public class TestCaseB extends TestCaseA
{
    @DataProvider(name = "data")
    private static Object[] createDataProvider()
    {
        return new Object[][] {{"b"},
        {"a"},
        {"b"},
        {"c"}};
    }

    @Factory(dataProvider = "data")
    public TestCaseB(String a)
    {
        System.out.println("Inside constructor with " + a + " parmeter and thread id "  + Thread.currentThread().getId());
    }

    @Test
    public void testA1() throws Exception
    {
        System.out.println("testA1 " + Thread.currentThread().getId());
    }

    @Test(dependsOnMethods = "testA1")
    public void testA2() throws Exception
    {
        System.out.println("testA2  with thread id " + Thread.currentThread().getId());
    }

    @Test(dependsOnMethods = "testA1")
    public void testB1() throws Exception
    {
        System.out.println("testB1  with thread id" + Thread.currentThread().getId());
    }
}

Answer 1 · 2024-03-28T12:43:26.000Z

Jdk8 - 7.5.1
Jdk11 - 7.9.0

Depending on ur jdk dependency please retry using above mentioned versions.

We cannot fix issues in earlier versions except for 7.9.0

Answer 2 · 2024-03-29T13:59:07.000Z

@galinaff - I retried this with TestNG 7.9.0 with a bit of tweaks to your sample so that the output is readable. I am not able to reproduce the problem. It works fine. I will be closing this. If this is still a problem with 7.9.0 please comment and we can re-open the issue.

Here's how the altered code looks like:

import org.testng.ITestContext;
import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;

import java.util.LinkedHashSet;
import java.util.Set;

public class TestCaseA {
    
    @BeforeClass(alwaysRun = true)
    protected void setUpTestClass(ITestContext context) {
        log();
    }

    @AfterClass(alwaysRun = true)
    protected void tearDownTestClass() {
        log();
    }

    protected void log() {
        ITestResult itr = Reporter.getCurrentTestResult();
        String instance = toString();
        long threadId = Thread.currentThread().getId();
        String text = String.format("Executing method [%s] on instance [%s] with thread id [%d]",
                itr.getMethod().getMethodName(), instance, threadId);
        Set<String> set = TestMain.logs.computeIfAbsent(instance, key -> new LinkedHashSet<>());
        synchronized (set) {
            set.add(text);
        }
    }
}

import org.testng.annotations.DataProvider;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;

public class TestCaseB extends TestCaseA {

    private final String text;

    @DataProvider(name = "data")
    private static Object[] createDataProvider() {
        return new Object[][]{{"b"},
                {"a"},
                {"b"},
                {"c"}};
    }

    @Factory(dataProvider = "data")
    public TestCaseB(String a) {
        this.text = a;
    }

    @Test
    public void testA1() {
        log();
    }

    @Test(dependsOnMethods = "testA1")
    public void testA2() {
        log();
    }

    @Test(dependsOnMethods = "testA1")
    public void testB1() {
        log();
    }

    @Override
    public String toString() {
        return "Instance[" + this.text + "]";
    }
}

import java.util.Collections;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;

import org.testng.TestNG;
import org.testng.xml.XmlClass;
import org.testng.xml.XmlSuite;
import org.testng.xml.XmlTest;

public class TestMain {
    public static final Map<String, Set<String>> logs = new ConcurrentHashMap<>();

    public static void main(String[] args) throws ClassNotFoundException {

        TestNG testNG = new TestNG();
        testNG.setVerbose(2);
        XmlSuite suite = newSuite();
        testNG.setXmlSuites(Collections.singletonList(suite));
        testNG.run();
        for (Map.Entry<String, Set<String>> each : logs.entrySet()) {
            System.err.println("Printing logs for instance " + each.getKey());
            each.getValue().forEach(System.err::println);
        }
    }

    private static XmlSuite newSuite() {
        XmlSuite suite = new XmlSuite();
        suite.setName("Suite");
        suite.setGroupByInstances(true);
        suite.setPreserveOrder(false);
        suite.setParallel(XmlSuite.ParallelMode.INSTANCES);
        suite.setThreadCount(2);
        addTestToSuite(suite);
        return suite;
    }

    private static void addTestToSuite(XmlSuite suite) {
        XmlTest test = new XmlTest();
        test.setName("testCases");
        test.setSuite(suite);
        test.setGroupByInstances(true);
        test.setPreserveOrder(false);
        test.setClasses(Collections.singletonList(new XmlClass(TestCaseB.class)));
        suite.setTests(Collections.singletonList(test));
    }
}

Execution output:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
PASSED: com.rationaleemotions.github.issue3105.TestCaseB.testA1
PASSED: com.rationaleemotions.github.issue3105.TestCaseB.testA1
PASSED: com.rationaleemotions.github.issue3105.TestCaseB.testB1
PASSED: com.rationaleemotions.github.issue3105.TestCaseB.testB1
PASSED: com.rationaleemotions.github.issue3105.TestCaseB.testA1
PASSED: com.rationaleemotions.github.issue3105.TestCaseB.testA2
PASSED: com.rationaleemotions.github.issue3105.TestCaseB.testA2
PASSED: com.rationaleemotions.github.issue3105.TestCaseB.testB1
PASSED: com.rationaleemotions.github.issue3105.TestCaseB.testA2
PASSED: com.rationaleemotions.github.issue3105.TestCaseB.testA2
PASSED: com.rationaleemotions.github.issue3105.TestCaseB.testA1
PASSED: com.rationaleemotions.github.issue3105.TestCaseB.testB1

===============================================
    testCases
    Tests run: 12, Failures: 0, Skips: 0
===============================================


===============================================
Suite
Total tests run: 12, Passes: 12, Failures: 0, Skips: 0
===============================================

Printing logs for instance Instance[a]
Executing method [setUpTestClass] on instance [Instance[a]] with thread id [26]
Executing method [testA1] on instance [Instance[a]] with thread id [26]
Executing method [testA2] on instance [Instance[a]] with thread id [26]
Executing method [testB1] on instance [Instance[a]] with thread id [26]
Executing method [tearDownTestClass] on instance [Instance[a]] with thread id [26]
Printing logs for instance Instance[b]
Executing method [setUpTestClass] on instance [Instance[b]] with thread id [27]
Executing method [testA1] on instance [Instance[b]] with thread id [27]
Executing method [setUpTestClass] on instance [Instance[b]] with thread id [26]
Executing method [testA1] on instance [Instance[b]] with thread id [26]
Executing method [testA2] on instance [Instance[b]] with thread id [27]
Executing method [testB1] on instance [Instance[b]] with thread id [27]
Executing method [tearDownTestClass] on instance [Instance[b]] with thread id [27]
Executing method [testA2] on instance [Instance[b]] with thread id [26]
Executing method [testB1] on instance [Instance[b]] with thread id [26]
Executing method [tearDownTestClass] on instance [Instance[b]] with thread id [26]
Printing logs for instance Instance[c]
Executing method [setUpTestClass] on instance [Instance[c]] with thread id [27]
Executing method [testA1] on instance [Instance[c]] with thread id [27]
Executing method [testA2] on instance [Instance[c]] with thread id [27]
Executing method [testB1] on instance [Instance[c]] with thread id [27]
Executing method [tearDownTestClass] on instance [Instance[c]] with thread id [27]

Process finished with exit code 0

Answer 3 · 2024-04-01T06:04:34.000Z

Thank you very much for your help!
Your example is working in the last version but still my example with System.out.println() still doesn't work. We still see the that thread continues to another class instance setup without finishing running the all the dependent methods and the after class.

Answer 4 · 2024-04-01T07:01:50.000Z

@galinaff - Feel free to refactor the sample (and perhaps add assertions as well), which will make the failure (deviation in terms of behavior expectations) evident. Print statements are a bit cumbersome to understand/follow when it comes to debugging problems. Please comment once the sample is available and we can relook at this issue again.

Answer 5 · 2024-04-02T09:44:50.000Z

Thank you!

We refactored the code in the following way, using System.out.println() or Reporter.log(), receiving the same result as you can see in the output. The main is the same for both examples.

Main

package tests.testCases;

import java.util.ArrayList;

import org.testng.TestNG;
import org.testng.xml.XmlClass;
import org.testng.xml.XmlSuite;
import org.testng.xml.XmlTest;

public class main
{
    public static void main(String[] args) throws ClassNotFoundException
    {
        XmlSuite suite;
        XmlTest test;
        TestNG testNG;
        XmlClass xmlClass;
        ArrayList<XmlTest> tests;
        ArrayList<XmlClass> testClasses;
        ArrayList<XmlSuite> testSuites;

        testNG = new TestNG();
        testNG.setVerbose(0);

        testSuites = new ArrayList<XmlSuite>();
        suite = new XmlSuite();
        suite.setName("Suite");

        test = new XmlTest();
        test.setName("testCases");
        test.setSuite(suite);
        test.setGroupByInstances(true);
        test.setPreserveOrder(false);

        testClasses = new ArrayList<XmlClass>();
        xmlClass = new XmlClass(Class.forName("tests.testCases.TestCaseB"));
        testClasses.add(xmlClass);
        test.setClasses(testClasses);

        tests = new ArrayList<XmlTest>();
        tests.add(test);

        suite.setTests(tests);
       suite.setGroupByInstances(true);
        suite.setPreserveOrder(false);
        suite.setParallel(XmlSuite.ParallelMode.INSTANCES);
        suite.setThreadCount(2);

        testSuites.add(suite);

        testNG.setXmlSuites(testSuites);

        testNG.run();
    }
}

1st test case sample

package tests.testCases;

import org.testng.ITestContext;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;

public class TestCaseA
{
    protected String text;

    @BeforeClass (alwaysRun = true)
    public void setUpTestClass(ITestContext context)
    {
        System.out.println(String.format("Executing method [setUpTestClass] on " +
                                          "instance [%s] with thread id [%d]",
                                          this.text,
                                          Thread.currentThread().getId()));
    }

    @AfterClass (alwaysRun = true)
    public void tearDownTestClass()
    {
        System.out.println(String.format("Executing method [tearDownTestClass] on " +
                                         "instance [%s] with thread id [%d]",
                                         this.text,
                                         Thread.currentThread().getId()));
    }
}

package tests.testCases;

import org.testng.ITestContext;
import org.testng.Reporter;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;

public class TestCaseA
{
    protected String text;

    @BeforeClass (alwaysRun = true)
    public void setUpTestClass(ITestContext context)
    {
        Reporter.log(String.format("Executing method [setUpTestClass] on instance " +
                                   "[%s] with thread id [%d]",
                                   this.text,
                                   Thread.currentThread().getId()),
                     true);
    }

    @AfterClass (alwaysRun = true)
    public void tearDownTestClass()
    {
        Reporter.log(String.format("Executing method [tearDownTestClass] on instance " +
                                   "[%s] with thread id [%d]",
                                   this.text,
                                   Thread.currentThread().getId()),
                     true);
    }
}

2nd test case example

package tests.testCases;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;

public class TestCaseB extends TestCaseA
{
    @DataProvider(name = "data")
    private static Object[] createDataProvider()
    {
        return new Object[][] {
        {"a"},
        {"b"},
        {"c"},
        {"d"}};
    }

    @Factory(dataProvider = "data")
    public TestCaseB(String a)
    {
        this.text = a;
    }

    @Test
    public void testA1() throws Exception
    {
        System.out.println(String.format("Executing method [testA1] on instance " +
                                         "[%s] with thread id [%d]",
                                         this.text,
                                         Thread.currentThread().getId()));
    }

    @Test(dependsOnMethods = "testA1")
    public void testA2() throws Exception
    {

        System.out.println(String.format("Executing method [testA2] on instance " +
                                         "[%s] with thread id [%d]",
                                         this.text,
                                         Thread.currentThread().getId()));
    }

    @Test(dependsOnMethods = "testA1")
    public void testB1() throws Exception
    {

        System.out.println(String.format("Executing method [testB1] on instance " +
                                         "[%s] with thread id [%d]",
                                         this.text,
                                         Thread.currentThread().getId()));
    }
}

package tests.testCases;

import org.testng.Reporter;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;

public class TestCaseB extends TestCaseA
{
    @DataProvider(name = "data")
    private static Object[] createDataProvider()
    {
        return new Object[][] {
        {"a"},
        {"b"},
        {"c"},
        {"d"}};
    }

    @Factory(dataProvider = "data")
    public TestCaseB(String a)
    {
        this.text = a;
    }

    @Test
    public void testA1() throws Exception
    {
        Reporter.log(String.format("Executing method [testA1] on instance " +
                                         "[%s] with thread id [%d]",
                                         this.text,
                                         Thread.currentThread().getId()),
                     true);
    }

    @Test(dependsOnMethods = "testA1")
    public void testA2() throws Exception
    {

        Reporter.log(String.format("Executing method [testA2] on instance [%s] with " +
                                   "thread id [%d]",
                                   this.text,
                                   Thread.currentThread().getId()),
                     true);
    }

    @Test(dependsOnMethods = "testA1")
    public void testB1() throws Exception
    {

        Reporter.log(String.format("Executing method [testB1] on instance [%s] with " +
                                   "thread id [%d]",
                                   this.text,
                                   Thread.currentThread().getId()) ,
                     true);
    }
}

The output for both cases is similar:

Executing method [setUpTestClass] on instance [d] with thread id [18]

Executing method [setUpTestClass] on instance [a] with thread id [19]

Executing method [testA1] on instance [a] with thread id [19]

Executing method [testA1] on instance [d] with thread id [18]

Executing method [setUpTestClass] on instance [b] with thread id [19]

Executing method [testA1] on instance [b] with thread id [19]

Executing method [setUpTestClass] on instance [c] with thread id [18]

Executing method [testA1] on instance [c] with thread id [18]

Executing method [testA2] on instance [a] with thread id [19]

Executing method [testB1] on instance [a] with thread id [19]

Executing method [testA2] on instance [d] with thread id [18]

Executing method [testB1] on instance [d] with thread id [18]

Executing method [tearDownTestClass] on instance [d] with thread id [18]

Executing method [testA2] on instance [c] with thread id [18]

Executing method [testB1] on instance [c] with thread id [18]

Executing method [tearDownTestClass] on instance [c] with thread id [18]

Executing method [tearDownTestClass] on instance [a] with thread id [19]

Executing method [testA2] on instance [b] with thread id [19]

Executing method [testB1] on instance [b] with thread id [19]

Executing method [tearDownTestClass] on instance [b] with thread id [19]

Answer 6 · 2024-04-02T12:41:23.000Z

@galinaff - I am kind of confused. So does that mean that the issue no longer exists or am I missing something?

Answer 7 · 2024-04-03T06:47:52.000Z

The issue still exists. From the output we can see that the threads continue to another setup test class instance without finishing to run all class's methods (A2, B1 and tearDownTestClass). For example, thread 18 runs the setupTestClass method and A1 test of instance [d] and then continues to setupTestClass method of instance [b] without running the A2, B2 and tearDownTestClass methods of instance [d].

Answer 8 · 2024-04-03T07:26:31.000Z

@galinaff

From the output we can see that the threads continue to another setup test class instance without finishing to run all class's methods (A2, B1 and tearDownTestClass).

You have set suite.setParallel(XmlSuite.ParallelMode.INSTANCES); So the instances will run in parallel. AFAIK, that's the behaviour.

Are you expecting that TestNG should do the following?

Create an instance of the test class via the factory
run its configuration and test
proceed with the next instance

If that's what you are expecting then you should be removing the parallel attribute and just stick to group-by-instances=true

Your test class also has depends which means that TestNG will segregate your execution into independent and dependent methods. TestNG will proceed with running your independent methods first, before it comes back to running the dependent methods.

Also can you please help with the following:

Point me to the documentation wherein this expectation is being called out. I ask because I dont remember this mentioned in the documentation.
Has this ever worked for you in any of the previous released testng versions?
Please help refactor the test case, to include assertions to match with what you have as expectations. Print statements don't help a lot to be honest and it becomes a bit more confusing in terms of ensuring that the desired behaviour is achieved.

Answer 9 · 2024-04-03T11:02:57.000Z

Yes, my expectation is exactly as you described:

Create an instance of the test class via the factory
Run its configuration and tests
Proceed with the next instance

And when executed in parallel, I expect the same flow executed by each one of the two threads, independently from each other. Or at least this is the way I interpret TestNG doc.

By this:
"Your test class also has depends which means that TestNG will segregate your execution into independent and dependent methods. TestNG will proceed with running your independent methods first, before it comes back to running the dependent methods."
Do you mean that these segregated methods will be executed in different class instances/different threads, no matter that they are contained in one class? Because my expectation is that the independent methods will be executed first, then the dependent ones but all this in the scope of one class instance execution.

Answer 10 · 2024-04-03T12:02:42.000Z

@galinaff

Create an instance of the test class via the factory
Run its configuration and tests
Proceed with the next instance

In that case, you should be running with a thread pool size of 1 because the moment you say parallel, TestNG will try to run things in parallel. In this case, your parallelism strategy is mentioned as instances, so TestNG will try to run the testng methods (configuration and test both) for each of the instances in parallel.

Do you mean that these segregated methods will be executed in different class instances/different threads, no matter that they are contained in one class?

No. TestNG will always associate the corresponding instance to the test/configuration method to which it belongs to. I believe that my earlier output should re-iterate this. But thread affinity is NOT guaranteed because, TestNG internally works with creating a Directed acyclic graph wherein, the independent methods are first found followed by the dependent ones (This is the same technique that even build tools also use, when it comes to figuring out what to run first and what to run later).

TestNG merely takes the responsibility of submitting task execution to a thread pool executor. After that the thread pool executor decides what to run on which thread based on the tasks that arrive via its linked blocking queue.

Because my expectation is that the independent methods will be executed first, then the dependent ones but all this in the scope of one class instance execution.

Yes, within the scope of class instantiation is true, but if you look at it, you have 2 threads, and you have 4 instances wherein there are configuration and also test methods that can be run.

So first the configuration methods will be run on each of the 2 threads (2 before class methods)
after that, we have 4 * 1 independent test methods (tests with no depends on) and so they will be run.
After that, the dependent methods in the context of its upstream dependency based on instance will get executed.

If you strictly want to go instance by instance execution, then you should reduce your thread pool size to 1 (or) disable parallel execution and just use group-by-instances to true (Oh btw, group-by-instances and parallel execution negate each other. So TestNG will NOT honour group-by-instances if you have turned on parallel execution)

I hope that clarifies how TestNG works.

Answer 11 · 2024-04-09T16:18:41.000Z

I understand, some changes in the code will have to be done then. Thank you very much for your explanation!