xUnit test runner doesn't respect parallel option under cog for test playlist

[framlingham]

We have been using this option in VS2022 17.7.6. I think it isn't working for us in our .NET Framework (.NET 4.8) solution for a test playlist.

But after staring at the little spinners, we tried updating our xunit.runner.json:

{
	"longRunningTestSeconds": 600,
	"methodDisplay": "method",
	"methodDisplayOptions": "all",
	"parallelizeAssembly": false,
	"parallelizeTestCollections": false
}

Those last two seem to do it just fine, but I would have expected the VS option to control this too. I tried updating all our xUnit NuGets, but no luck:

• xunit 2.6.2 (at the same version are xunit.assert, xunit.core, xunit.extensibility.core, xunit.runner.console)
• xunit.runner.visualstudio 2.5.4
• xunit.abstractions 2.0.3
• xunit.analyzers 1.6.0

[framlingham]

I checked out similar issues, I can't find anything that seems to match or they are a lot older:

• #185
• #59
• #191
• #156

[bradwilson]

Can you provide a repro? Is this a single project or multiple projects? What is the behavior you're expecting vs. getting? Which xunit.runner.json options are you changing, and what result does that have?

If you've read through #191 you'd see that we have no control over how multi-assembly parallelism is executed in VSTest. We can instruct people on how to invoke dotnet test vs. dotnet vstest to try to simulate some parallelism options, but it's not clear to me what that menu option is intended to do.

[framlingham]

Also merry Christmas! No rush on this issue since we've worked around it.

[framlingham]

Sorry, I didn't get to your detailed response. So attempting to read it properly:

Behavior	dotnet test	dotnet vstest	VS2022 UI
Run all tests in parallel	yes, forced default by processes	yes, --parallel and list all assemblies	yes, default
Run all tests in series	separate commands (per project)	yes, list all assemblies	done with mouse, one at a time
Run subset	nope	yes	yes

But I get the impression that not much of this is controllable by xUnit or visualstudio.xunit. Who owns what so I can pester the right people/orgs?

[framlingham]

I'll see what I can do about a repro. It is unit tests within same assembly/project. I've tried xunit.runner.json with and without the below two lines:

"parallelizeAssembly": false,
"parallelizeTestCollections": false

It does what it says - what I'd like to try to do is to get the "Run Tests In Parallel" option I screen grabbed earlier to do the same as this. I get the impression that I might need to put this request to another organisation? If so, could you direct me please? I'm not sure who owns what, so please forgive me if I got it wrong.

[framlingham]

OK, so I tested these combinations. The first column mentiones whether the two json lines were included that specify sequential for assembly and test collections. I only ran the one test project/assembly:

xunit.runner.json	UI parallel selected	Result
with 2 lines	parallel	Ran sequentially as specified in xunit.runner.json, finished in 5min
with 2 lines	sequential	Ran sequentially as specified in xunit.runner.json, finished in 5min
without 2 lines	parallel	Ran parallel as specified in xunit.runner.json, stalled around 5min, stopped a bit later manually
without 2 lines	sequential	Ran sequentially as specified in xunit.runner.json, stalled around 5min, it aborted at about 9 minutes

It looks like it would be nice if whoever wrote the UI part had some logic that hides the setting (or a hint that an xunit.runner.json is defined) or adds an option to do what the xunit.runner.json says.

[bradwilson]

I don't think there's anything we can do about this.

It's a well known "feature" of VSTest that the way they invoke us completely ignores the parallelizeAssembly setting that we have, so they're completely in control here and we have no say at all.

As for parallelizeTestCollections, that's definitely always in our control and should be independent of whatever you choose with the cog.

Who owns what so I can pester the right people/orgs?

VSTest: https://github.com/microsoft/vstest/

You can bug all you want, but I can pretty much guarantee you they're going to tell you it's working as they want it to work. (Not to say that you shouldn't try, just trying not to get your hopes up.)