aspnet/Testing

Some ConditionalTheory tests are run N^2 times

Closed this issue · 3 comments

cesarblum commented

For example, in Kestrel:

> dotnet test --filter FullyQualifiedName=Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads
Build started, please wait...
C:\Users\cesars\AppData\Local\Microsoft\dotnet\sdk\1.0.0-rc4-004757\Microsoft.Common.CurrentVersion.targets(1952,5): warning MSB3277: Found conflicts between different versions of the same dependent assembly that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed. [D:\src\aspnet\KestrelHttpServer\test\Microsoft.AspNetCore.Server.Kestrel.FunctionalTests\Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.csproj]
C:\Users\cesars\AppData\Local\Microsoft\dotnet\sdk\1.0.0-rc4-004757\Microsoft.Common.CurrentVersion.targets(1952,5): warning MSB3277: Found conflicts between different versions of the same dependent assembly that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed. [D:\src\aspnet\KestrelHttpServer\test\Microsoft.AspNetCore.Server.Kestrel.FunctionalTests\Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.csproj]
Build completed.

Test run for D:\src\aspnet\KestrelHttpServer\test\Microsoft.AspNetCore.Server.Kestrel.FunctionalTests\bin\Debug\netcoreapp1.1\Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.dll(.NETCoreApp,Version=v1.1)
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 6)(threadCount: 1)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 6)(threadCount: 2)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 6)(threadCount: 3)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 6)(threadCount: 4)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 6)(threadCount: 5)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 6)(threadCount: 6)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 6)(threadCount: 7)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 6)(threadCount: 8)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 6)(threadCount: 9)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 6)(threadCount: 10)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 4)(threadCount: 1)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 4)(threadCount: 2)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 4)(threadCount: 3)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 4)(threadCount: 4)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 4)(threadCount: 5)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 4)(threadCount: 6)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 4)(threadCount: 7)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 4)(threadCount: 8)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 4)(threadCount: 9)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 4)(threadCount: 10)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 8)(threadCount: 1)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 8)(threadCount: 2)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 8)(threadCount: 3)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 8)(threadCount: 4)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 8)(threadCount: 5)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 8)(threadCount: 6)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 8)(threadCount: 7)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 8)(threadCount: 8)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 8)(threadCount: 9)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 8)(threadCount: 10)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 9)(threadCount: 1)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 9)(threadCount: 2)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 9)(threadCount: 3)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 9)(threadCount: 4)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 9)(threadCount: 5)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 9)(threadCount: 6)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 9)(threadCount: 7)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 9)(threadCount: 8)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 9)(threadCount: 9)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 9)(threadCount: 10)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 3)(threadCount: 1)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 3)(threadCount: 2)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 3)(threadCount: 3)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 3)(threadCount: 4)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 3)(threadCount: 5)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 3)(threadCount: 6)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 3)(threadCount: 7)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 3)(threadCount: 8)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 3)(threadCount: 9)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 3)(threadCount: 10)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 1)(threadCount: 1)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 1)(threadCount: 2)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 1)(threadCount: 3)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 1)(threadCount: 4)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 1)(threadCount: 5)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 1)(threadCount: 6)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 1)(threadCount: 7)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 1)(threadCount: 8)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 1)(threadCount: 9)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 1)(threadCount: 10)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 10)(threadCount: 1)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 10)(threadCount: 2)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 10)(threadCount: 3)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 10)(threadCount: 4)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 10)(threadCount: 5)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 10)(threadCount: 6)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 10)(threadCount: 7)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 10)(threadCount: 8)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 10)(threadCount: 9)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 10)(threadCount: 10)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 7)(threadCount: 1)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 7)(threadCount: 2)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 7)(threadCount: 3)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 7)(threadCount: 4)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 7)(threadCount: 5)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 7)(threadCount: 6)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 7)(threadCount: 7)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 7)(threadCount: 8)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 7)(threadCount: 9)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 7)(threadCount: 10)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 2)(threadCount: 1)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 2)(threadCount: 2)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 2)(threadCount: 3)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 2)(threadCount: 4)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 2)(threadCount: 5)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 2)(threadCount: 6)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 2)(threadCount: 7)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 2)(threadCount: 8)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 2)(threadCount: 9)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 2)(threadCount: 10)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 5)(threadCount: 1)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 5)(threadCount: 2)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 5)(threadCount: 3)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 5)(threadCount: 4)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 5)(threadCount: 5)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 5)(threadCount: 6)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 5)(threadCount: 7)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 5)(threadCount: 8)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 5)(threadCount: 9)
Passed   Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ThreadCountTests.OneToTenThreads(threadCount: 5)(threadCount: 10)

Total tests: 100. Passed: 100. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.7932 Minutes

Must be an issue with our ConditionalTheory, since it doesn't repro when I change it to Theory.

cc @pranavkm @NTaylorMullen @rynowak

cesarblum commented

Small repro:

[InlineData(0)]
[InlineData(1)]
public void MyTest(int arg)
{
    Assert.True(true);
}

Output:

Starting test execution, please wait...
Passed   Repro.MyClass.MyTest(arg: 0)(arg: 0)
Passed   Repro.MyClass.MyTest(arg: 0)(arg: 1)
Passed   Repro.MyClass.MyTest(arg: 1)(arg: 0)
Passed   Repro.MyClass.MyTest(arg: 1)(arg: 1)
NTaylorMullen commented

Obviously just trying to make sure things really pass :trollface:

  • 😄1
cesarblum commented

I think I know what the problem is. In

Testing/src/Microsoft.AspNetCore.Testing/xunit/SkipReasonTestCase.cs

Line 113 in 3714e54

runner = new XunitTheoryTestCaseRunner(
we always use XunitTheoryTestCaseRunner to run theories. But that class should only be used for theories of type XunitTheoryTestCase that could not be enumerated prior to test execution, so they will be enumerated upon execution (if I understand correctly, that's because their arguments could not be serialized, but ¯\(ツ)/¯). So what's happening here is that we're enumerating those theories prior to test execution and then again during execution, resulting in the behavior above.

I think the fix is to keep the type of the test cases discovered by ConditionalAttributeDiscoverer's inner discoverer, and only use an XunitTheoryTestCaseRunner in SkipReasonTestCase.RunAsync() if the test type is XunitTheoryTestCase (theories can be of type XunitTestCase - https://github.com/xunit/xunit/blob/c0e29e9d3efb020a1670a9a6eef3c9a47e944fa4/src/xunit.execution/Sdk/Frameworks/TheoryDiscoverer.cs).