Dependent methods not running in correct order when using parallel instances

[mihalyr]

TestNG version 7.3.0

Dependent methods might not always execute in the correct order with parallel=instances, below is a reproducer, which quite often fails (not 100% reliable as sometimes does succeed too). Not sure why, but I could not reproduce it without the added delay. Also the number of test methods in the class had a relation to failures too.

Is this a known issue? It seems I cannot use parallel=instances with dependent methods even if my code is thread safe if there are dependencies between test methods.

import org.testng.annotations.BeforeClass;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;

import java.util.concurrent.atomic.AtomicInteger;

import static org.testng.Assert.assertEquals;

@Test(suiteName = "testParallel")
public class TestNGParallelDependent {
    private final String prefix;
    private final int delay;
    private final AtomicInteger counter = new AtomicInteger();

    private volatile String value;

    @Factory(dataProvider = "dp")
    TestNGParallelDependent(String prefix, int delay) {
        this.prefix = prefix;
        this.delay = delay;
    }

    @DataProvider
    static Object[][] dp() {
        int instances = 3;
        int delay = 1;
        Object[][] params = new Object[instances][2];
        for (int i = 0; i < instances; i++) {
            params[i][0] = "v" + (i + 1);
            params[i][1] = delay * 1000;
            delay <<= 1;
        }
        return params;
    }

    @BeforeClass
    void setup() {
        work(delay);
    }

    @Test
    void testA() {
        assertEquals(1, counter.incrementAndGet());
    }

    @Test(dependsOnMethods = "testA")
    void testB() {
        assertEquals(2, counter.incrementAndGet());
    }

    @Test(dependsOnMethods = "testB")
    void testC() {
        assertEquals(3, counter.incrementAndGet());
    }

    @Test(dependsOnMethods = "testC")
    void testD() {
        assertEquals(4, counter.incrementAndGet());
    }

    @Test(dependsOnMethods = "testD")
    void testE() {
        assertEquals(5, counter.incrementAndGet());
    }

    @Test(dependsOnMethods = "testE")
    void testF() {
        assertEquals(6, counter.incrementAndGet());
    }

    @Test(dependsOnMethods = "testF")
    void testG() {
        assertEquals(7, counter.incrementAndGet());
    }

    @Test(dependsOnMethods = "testG")
    void testH() {
        assertEquals(8, counter.incrementAndGet());
    }

    @Test(dependsOnMethods = "testH")
    void testI() {
        assertEquals(9, counter.incrementAndGet());
    }

    @Test(dependsOnMethods = "testI")
    void testJ() {
        assertEquals(10, counter.incrementAndGet());
    }

    private static void work(int delay) {
        try {
            Thread.sleep(delay);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
    }
}

Originally posted by @mihalyr in #751 (comment)

[krmahadevan]

@mihalyr - Can you please edit this issue and include the TestNG version information as well (if you aren't using 7.3.0)

[mihalyr]

Yep, this is on 7.3.0 also, added to the description

[mihalyr]

Test code also here with config

[krmahadevan]

@mihalyr - This is fixed as part of #2321. This problem should go away when you use -Dtestng.thread.affinity=true and then run a suite that contains this class when running parallel instances.

@juherr - Considering the fact that the current TestNG design will not let a user run dependent methods in the same thread as the independent thread (without using -Dtestng.thread.affinity=true which only works as long there's only one downstream dependent method for every independent method), what do you suggest that we do in this case ?

[juherr]

@krmahadevan I suggest that we keep the issue open and update the documentation in order to warn about the current limitation.

[mihalyr]

@krmahadevan Thanks for the update, which version will include the changes you mentioned, 7.4.0?

[krmahadevan]

Yes 7.4.0 will contain the changes

[ristoporila]

@krmahadevan , when can we expect 7.4.0 to be released? I am struggling with this as well.

[krmahadevan]

@ristoporila - Its going to take some time since before 7.4.0 gets released.

@cbeust @juherr so far we have only around 8 bug fixes that have landed in master. Do we go ahead with a release ?

[juherr]

@krmahadevan 👍

[BrandonDudek]

This is still a bug in v7.8.0.

[krmahadevan]

@BrandonDudek - Do you have a sample that can be used to reproduce the problem ?