dotnet test doesn't work on .NET Core 3.0 Preview 6
livarcocc opened this issue · 22 comments
From @omajid on Friday, June 28, 2019 9:56:30 PM
Steps to reproduce
dotnet --info
dotnet new xunit
dotnet test
Expected behavior
The generated UnitTest1 class with the single Test1 is executed and the test is marked as passed
Actual behavior
Microsoft (R) Test Execution Command Line Tool Version 16.0.1-dev+ea406627f919daa1d8da7daabe2d1f6619d2ad72
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
No test is available in /home/omajid/temp/test/bin/Debug/netcoreapp3.0/test.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
Additionally, path to test adapters can be specified using /TestAdapterPath command. Example /TestAdapterPath:<pathToCustomAdapters>.
Environment data
dotnet --info output:
.NET Core SDK (reflecting any global.json):
Version: 3.0.100-preview6-012264
Commit: be3f0c1a03
Runtime Environment:
OS Name: fedora
OS Version: 30
OS Platform: Linux
d RID: fedora.30-x64
Base Path: /usr/lib64/dotnet/sdk/3.0.100-preview6-012264/
Host (useful for support):
Version: 3.0.0-preview6-27804-01
Commit: 17dd7897d6
.NET Core SDKs installed:
3.0.100-preview6-012264 [/usr/lib64/dotnet/sdk]
.NET Core runtimes installed:
Microsoft.NETCore.App 3.0.0-preview6-27804-01 [/usr/lib64/dotnet/shared/Microsoft.NETCore.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download
This dotnet SDK was compiled from the v3.0.0-preview6-27804-01 tag of source-build.
Copied from original issue: dotnet/cli#11663
@omajid I just tried this on fresh machine, & things worked as expected. Can you please try it on some another box & let us know if you are facing similar issue
Also you can enable diagnostics logs via https://github.com/microsoft/vstest-docs/blob/master/docs/diagnose.md, & share those with us.
You can also try running dotnet new mstest, & verifying whether tests for those are run or not.
I also having problems using xunit. I can see that you also run Fedora 30, so maybe a dotnet-sig problem? Tried mstest as suggested by @mayankbansal018 and no received no errors.
Anyways here are environmental info:
`.NET Core SDK (reflecting any global.json):
Version: 3.0.100-preview6-012264
Commit: be3f0c1a03
Runtime Environment:
OS Name: fedora
OS Version: 30
OS Platform: Linux
RID: fedora.30-x64
Base Path: /usr/lib64/dotnet/sdk/3.0.100-preview6-012264/
Host (useful for support):
Version: 3.0.0-preview6-27804-01
Commit: N/A
.NET Core SDKs installed:
2.1.403 [/usr/lib64/dotnet/sdk]
2.1.507 [/usr/lib64/dotnet/sdk]
2.2.107 [/usr/lib64/dotnet/sdk]
3.0.100-preview6-012264 [/usr/lib64/dotnet/sdk]
.NET Core runtimes installed:
Microsoft.NETCore.App 2.1.11 [/usr/lib64/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.5 [/usr/lib64/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.0.0-preview6-27804-01 [/usr/lib64/dotnet/shared/Microsoft.NETCore.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download
`
And here is the result of dotnet test --diag:log.txt
https://pastebin.com/3J0TLEhM
@mayankbansal018 Sorry about that. This looks like a source-build issue.
dotnet new xunit && dotnet test
This works with Microsoft-built .NET Core 3.0 Preview 6 but not with self-built .NET Core using source-build.
dotnet new mstest && dotnet test works with both.
Can someone move this issue to the source-build repo? Otherwise I will close this and open one there myself.
Sorry for the noise.
so maybe a dotnet-sig problem?
Yeah. The dotnet-sig packages use source-build to build and assemble .NET Core. So they should have the exact same set of issues.
I also having problems using xunit
For a workaround, you can disable the @dotnet-sig/dotnet-preview copr repo and downgrade to use the 2.1 or 2.2 SDK which should not have these issues. Obviously this is not great advice if you enabled @dotnet-sig/dotnet-preview intentionally to identify source-build specific issues.
No test is available in /home/omajid/temp/test/bin/Debug/netcoreapp3.0/test.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
With this error, dotnet test is returning a zero (that is: success) exit code.
Here's the log file from dotnet new xunit && dotnet test --diag:log.txt source-build-log.txt
I've been investigating this a bit.
So with source-build .NET Core:
$ dotnet new xunit -o xunit_sb
$ cd xunit_sb/
$ dotnet test
Test run for /tmp/xunit_sb/bin/Debug/netcoreapp3.0/xunit_sb.dll(.NETCoreApp,Version=v3.0)
Microsoft (R) Test Execution Command Line Tool Version 16.0.1-dev+ea406627f919daa1d8da7daabe2d1f6619d2ad72
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
No test is available in /tmp/xunit_sb/bin/Debug/netcoreapp3.0/xunit_sb.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
Additionally, path to test adapters can be specified using /TestAdapterPath command. Example /TestAdapterPath:<pathToCustomAdapters>.
with Microsoft built .NET Core:
$ dotnet new xunit -o xunit_ms
$ cd xunit_ms
$ dotnet test
Test run for /tmp/xunit_ms/bin/Debug/netcoreapp3.0/xunit_ms.dll(.NETCoreApp,Version=v3.0)
Microsoft (R) Test Execution Command Line Tool Version 16.0.1
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.1340 Seconds
ok, MS works, source-build doesn't.
now, let's copy the output of the xunit_ms folder into the xunit_sb folder:
$ cp -r /tmp/xunit_ms/bin/Debug/netcoreapp3.0/* bin/Debug/netcoreapp3.0/
using the source-build dotnet, testing both:
$ dotnet exec /usr/lib64/dotnet/sdk/3.0.100-preview6-012264/vstest.console.dll --framework:.NETCoreApp,Version=v3.0 /tmp/xunit_sb/bin/Debug/netcoreapp3.0/xunit_ms.dll
Microsoft (R) Test Execution Command Line Tool Version 16.0.1-dev+ea406627f919daa1d8da7daabe2d1f6619d2ad72
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.6217 Seconds
$ dotnet exec /usr/lib64/dotnet/sdk/3.0.100-preview6-012264/vstest.console.dll --framework:.NETCoreApp,Version=v3.0 /tmp/xunit_sb/bin/Debug/netcoreapp3.0/xunit_sb.dll
Microsoft (R) Test Execution Command Line Tool Version 16.0.1-dev+ea406627f919daa1d8da7daabe2d1f6619d2ad72
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
No test is available in /tmp/xunit_sb/bin/Debug/netcoreapp3.0/xunit_sb.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
Additionally, path to test adapters can be specified using /TestAdapterPath command. Example /TestAdapterPath:<pathToCustomAdapters>.
This is weird. The xunit dll that was build using Microsoft dotnet works, but the one compiled with source-build dotnet doesn't.
Let's use the deps.json from xunit_ms and replace xunit_ms with xunit_sb:
$ sed 's/xunit_ms/xunit_sb/g' xunit_ms.deps.json >xunit_sb.deps.json
$ $ dotnet exec /usr/lib64/dotnet/sdk/3.0.100-preview6-012264/vstest.console.dll --framework:.NETCoreApp,Version=v3.0 /tmp/xunit_sb/bin/Debug/netcoreapp3.0/xunit_sb.dll
Microsoft (R) Test Execution Command Line Tool Version 16.0.1-dev+ea406627f919daa1d8da7daabe2d1f6619d2ad72
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.6480 Seconds
ok, it's working now.
What's the difference between the deps.json files?
$ diff -u xunit_sb.deps.json xunit_ms.deps.json
--- xunit_sb.deps.json 2019-07-03 09:43:56.920221598 +0200
+++ xunit_ms.deps.json 2019-07-03 09:30:11.838834146 +0200
@@ -6,7 +6,7 @@
"compilationOptions": {},
"targets": {
".NETCoreApp,Version=v3.0": {
- "xunit_sb/1.0.0": {
+ "xunit_ms/1.0.0": {
"dependencies": {
"Microsoft.NET.Test.Sdk": "16.0.1",
"Microsoft.NETCore.App": "3.0.0-preview6-27804-01",
@@ -14,7 +14,7 @@
"xunit.runner.visualstudio": "2.4.0"
},
"runtime": {
- "xunit_sb.dll": {}
+ "xunit_ms.dll": {}
}
},
"Microsoft.CodeCoverage/16.0.1": {
@@ -1383,7 +1383,7 @@
}
},
"libraries": {
- "xunit_sb/1.0.0": {
+ "xunit_ms/1.0.0": {
"type": "project",
"serviceable": false,
"sha512": ""
@@ -1398,49 +1398,49 @@
"Microsoft.CSharp/4.0.1": {
"type": "package",
"serviceable": true,
- "sha512": "sha512-17h8b5mXa87XYKrrVqdgZ38JefSUqLChUQpXgSnpzsM0nDOhE40FTeNWOJ/YmySGV6tG6T8\u002bhjz6vxbknHJr6A==",
+ "sha512": "sha512-17h8b5mXa87XYKrrVqdgZ38JefSUqLChUQpXgSnpzsM0nDOhE40FTeNWOJ/YmySGV6tG6T8+hjz6vxbknHJr6A==",
"path": "microsoft.csharp/4.0.1",
"hashPath": "microsoft.csharp.4.0.1.nupkg.sha512"
},
"Microsoft.DotNet.PlatformAbstractions/1.0.3": {
"type": "package",
"serviceable": true,
- "sha512": "sha512-rF92Gp5L2asYrFNf0cKNBxzzGLh1krHuj6TRDk9wdjN2qdvJLaNYOn1s9oYkMlptYX436KiEFqxhLB\u002bI5veXvQ==",
+ "sha512": "sha512-rF92Gp5L2asYrFNf0cKNBxzzGLh1krHuj6TRDk9wdjN2qdvJLaNYOn1s9oYkMlptYX436KiEFqxhLB+I5veXvQ==",
"path": "microsoft.dotnet.platformabstractions/1.0.3",
"hashPath": "microsoft.dotnet.platformabstractions.1.0.3.nupkg.sha512"
},
"Microsoft.Extensions.DependencyModel/1.0.3": {
"type": "package",
"serviceable": true,
- "sha512": "sha512-Z3o19EnheuegmvgpCzwoSlnCWxYA6qIUhvKJ7ifKHHvU7U\u002boYR/gliLiL3LVYOOeGMEEzkpJ5W67sOcXizGtlw==",
+ "sha512": "sha512-Z3o19EnheuegmvgpCzwoSlnCWxYA6qIUhvKJ7ifKHHvU7U+oYR/gliLiL3LVYOOeGMEEzkpJ5W67sOcXizGtlw==",
"path": "microsoft.extensions.dependencymodel/1.0.3",
"hashPath": "microsoft.extensions.dependencymodel.1.0.3.nupkg.sha512"
},
"Microsoft.NET.Test.Sdk/16.0.1": {
"type": "package",
"serviceable": true,
- "sha512": "sha512-ON7UIMIhAwrYb0ep\u002b3ztoWVGvtfo88IhiHVnbyOiuVsi8bOMCdMPVcR\u002bEX1WYGgKAd030pHRaxazMlkQ6uDyJQ==",
+ "sha512": "sha512-ON7UIMIhAwrYb0ep+3ztoWVGvtfo88IhiHVnbyOiuVsi8bOMCdMPVcR+EX1WYGgKAd030pHRaxazMlkQ6uDyJQ==",
"path": "microsoft.net.test.sdk/16.0.1",
"hashPath": "microsoft.net.test.sdk.16.0.1.nupkg.sha512"
},
"Microsoft.NETCore.App/3.0.0-preview6-27804-01": {
"type": "package",
"serviceable": true,
- "sha512": "sha512-ZusFeFw4lm4f3W1GyaTDbEdqln8N0fWFijKpykPLLvrEEsppfZwx8n/UeMm/e56QspzGHKpJfmKi4YF2Z6\u002bG6A==",
+ "sha512": "sha512-ZusFeFw4lm4f3W1GyaTDbEdqln8N0fWFijKpykPLLvrEEsppfZwx8n/UeMm/e56QspzGHKpJfmKi4YF2Z6+G6A==",
"path": "microsoft.netcore.app/3.0.0-preview6-27804-01",
"hashPath": "microsoft.netcore.app.3.0.0-preview6-27804-01.nupkg.sha512"
},
"Microsoft.NETCore.Platforms/3.0.0-preview6.19303.8": {
"type": "package",
"serviceable": true,
- "sha512": "sha512-9nE9InmtkWL1ZIAvzZPoe8zzr8JNrmyofR5\u002b\u002b0NCLomrLSbR70b60hLXFE/7vXIVgbKdTo3CTsCcFTOSZoOIEw==",
+ "sha512": "sha512-9nE9InmtkWL1ZIAvzZPoe8zzr8JNrmyofR5++0NCLomrLSbR70b60hLXFE/7vXIVgbKdTo3CTsCcFTOSZoOIEw==",
"path": "microsoft.netcore.platforms/3.0.0-preview6.19303.8",
...
-> mostly sha512 values.
Looking at the diag host log:
With xunit_ms:
TpTrace Verbose: 0 : 8395, 4, 2019/07/03, 09:50:11.342, 1861530083800, testhost.dll, BaseRunTests.RunTestInternalWithExecutors: Running tests for executor://xunit/VsTestRunner2/netcoreapp
TpTrace Information: 0 : 8395, 4, 2019/07/03, 09:50:11.350, 1861538056332, testhost.dll, [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0 (64-bit .NET Core 3.0.0-preview6-27804-01)
TpTrace Verbose: 0 : 8395, 4, 2019/07/03, 09:50:11.350, 1861538160539, testhost.dll, TestRequestHandler.SendData: sending data from testhost: {"Version":2,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0 (64-bit .NET Core 3.0.0-preview6-27804-01)"}}
TpTrace Information: 0 : 8395, 4, 2019/07/03, 09:50:11.716, 1861903553968, testhost.dll, AssemblyResolver.OnResolve: xunit_sb: Resolving assembly.
TpTrace Information: 0 : 8395, 4, 2019/07/03, 09:50:11.716, 1861903786858, testhost.dll, AssemblyResolver.OnResolve: Resolved assembly: xunit_sb, from path: /tmp/xunit_sb/bin/Debug/netcoreapp3.0/xunit_sb.dll
TpTrace Information: 0 : 8395, 4, 2019/07/03, 09:50:11.716, 1861903905683, testhost.dll, AssemblyResolver.OnResolve: xunit.runner.reporters.netcoreapp10: Resolving assembly.
TpTrace Information: 0 : 8395, 4, 2019/07/03, 09:50:11.716, 1861904143923, testhost.dll, AssemblyResolver.OnResolve: Resolved assembly: xunit.runner.reporters.netcoreapp10, from path: /tmp/xunit_sb/bin/Debug/netcoreapp3.0/xunit.runner.reporters.netcoreapp10.dll
With xunit_sb:
TpTrace Verbose: 0 : 8258, 4, 2019/07/03, 09:48:56.445, 1786632628833, testhost.dll, BaseRunTests.RunTestInternalWithExecutors: Running tests for executor://xunit/VsTestRunner2/netcoreapp
TpTrace Information: 0 : 8258, 4, 2019/07/03, 09:48:56.453, 1786640757731, testhost.dll, [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0 (64-bit .NET Core 3.0.0-preview6-27804-01)
TpTrace Verbose: 0 : 8258, 4, 2019/07/03, 09:48:56.453, 1786640877978, testhost.dll, TestRequestHandler.SendData: sending data from testhost: {"Version":2,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0 (64-bit .NET Core 3.0.0-preview6-27804-01)"}}
TpTrace Verbose: 0 : 8258, 4, 2019/07/03, 09:48:56.523, 1786710813462, testhost.dll, BaseRunTests.RunTestInternalWithExecutors: Completed running tests for executor://xunit/VsTestRunner2/netcoreapp
The difference in deps.json files is causing the xunit_ms to resolve assemblies, while the xunit_sb doesn't.
@tmds , thanks for sharing this analysis, I suppose you can log this issue at xUnit repo, & they can provide a fix, or reasoning.
Let me know if this can be closed.
@mayankbansal018 I'm not sure the issue is with xunit, I can't derive that from the logging.
There is a difference between Microsoft and source-build .NET Core, so I think this needs a fix within one of the .NET Core repos.
I had copied over a bunch of files from xunit_ms:
$ cp -r /tmp/xunit_ms/bin/Debug/netcoreapp3.0/* bin/Debug/netcoreapp3.0/
The xunit_ms dll also runs successfully with the source-build dotnet when only copying over: xunit_ms.deps.json, xunit_ms.dll, xunit_ms.pdb, xunit_ms.runtimeconfig.dev.json, xunit_ms.runtimeconfig.json.
@mayankbansal018 do you have some thoughts on how the deps.json file could be affecting the resolved assemblies for tests?
@tmds AFAIK dotnet uses it resolve dependencies for an assembly. https://natemcmaster.com/blog/2017/12/21/netcore-primitives/#depsjson should help.
My two cents: the hashes are actually the same, since \u002b is +. I guess some library involved in source-build nuget restore or deps file generation likes to escape characters more. This leaves the diff as:
--- xunit_sb.deps.json 2019-07-03 09:43:56.920221598 +0200
+++ xunit_ms.deps.json 2019-07-03 09:30:11.838834146 +0200
@@ -6,7 +6,7 @@
"compilationOptions": {},
"targets": {
".NETCoreApp,Version=v3.0": {
- "xunit_sb/1.0.0": {
+ "xunit_ms/1.0.0": {
"dependencies": {
"Microsoft.NET.Test.Sdk": "16.0.1",
"Microsoft.NETCore.App": "3.0.0-preview6-27804-01",
@@ -14,7 +14,7 @@
"xunit.runner.visualstudio": "2.4.0"
},
"runtime": {
- "xunit_sb.dll": {}
+ "xunit_ms.dll": {}
}
},
"Microsoft.CodeCoverage/16.0.1": {
@@ -1383,7 +1383,7 @@
}
},
"libraries": {
- "xunit_sb/1.0.0": {
+ "xunit_ms/1.0.0": {
"type": "project",
"serviceable": false,
"sha512": ""I don't have ready access to a 3.0 Preview 6 source-built SDK (my focus is dotnet/core-setup right now), is it possible to share your repro bits? (Even better if it runs standalone in a Docker container.)
You could consider decompiling the xunit DLLs and diffing the results. I used ILSpy to do that in the past to figure out what was different about a source-built mono/linker tool vs. the prebuilt.
@dagood Here's a Dockerfile that should reproduce the issue:
FROM fedora:30
RUN dnf install dnf-plugins-core -y && \
dnf copr enable @dotnet-sig/dotnet-preview -y && \
dnf install dotnet-sdk-3.0 -y
CMD dotnet --info && \
dotnet new xunit -o Sample && \
cd Sample && \
dotnet testI repro'd, and also ran a Msft dotnet on the "bad" bin dir in the same container, and that repro'd the bad behavior. Based on this, I think it's (even more) safe to focus on the deps file as the cause.
I don't know how to figure out why whatever's using the deps file isn't interpreting \u002b as +.
On the deps file creation side, I know a little more. The Msft SDK carries an old Microsoft.Extensions.DependencyModel 2.2, but the source-built SDK carries DependencyModel 3.0. (I imagine it's forced up via package version props, as expected.) DependencyModel recently started using System.Text.Json instead of Newtonsoft.Json. It looks like this "feature" in System.Text.Json is causing the deps file + => \u002b diff: https://github.com/dotnet/corefx/issues/38354.
So, two issues:
- Wherever the deps file is consumed isn't handling utf-escaped chars.
- Deps file production uses System.Text.Json, which escapes
+when it doesn't need to.- Maybe this just needs to set a non-default escaping behavior.
@ericstj, what do you think? Do you know how to do (2), so maybe Red Hat could apply it as a patch for now?
The deps file is consumed by the host. cc @vitek-karas
the hashes are actually the same, since \u002b is +
Aha! I hadn't noticed. It was so weird that using 'different' hashes caused things to work, but in fact, they were not so different 😄
It's easy to manually repro this on Windows too, by the way (if you have 3.0.100-preview6-012264 installed):
dotnet new xunit -o repro ; cd repro
dotnet test
(gc .\bin\Debug\netcoreapp3.0\repro.deps.json) | %{ $_.Replace('+', '\u002b') } | sc .\bin\Debug\netcoreapp3.0\repro.deps.json
dotnet exec "C:\Program Files\dotnet\sdk\3.0.100-preview6-012264\vstest.console.dll" "--framework:.NETCoreApp,Version=v3.0" .\bin\Debug\netcoreapp3.0\repro.dll@vitek-karas took a look at corehost logs I sent offline (COREHOST_TRACE=1, COREHOST_TRACEFILE=t.log) but he didn't see anything to indicate the host is involved in this. If there were a problem there, a hard failure would be expected anyway, not just failing to find extensions. We think there's probably some xunit or vstest code trying to find test adapters, also trying to parse this file for some reason.
I tried to look at the vstest code to figure out where it goes wrong, but no luck. The idea was that due to the change of JSON parsing library in Microsoft.Extensions.DependencyModel, the old version would have trouble parsing the new json file - but it seems to work OK (vstest does indeed use the DependencyModel APIs to parse the .deps.json).
I think at this point it needs somebody more familiar with the vstest codebase to debug through it.
I've looked into this further and traced down the issue to xunit json parsing: xunit/xunit#1980.
See https://github.com/dotnet/core-setup/issues/7137
The reason why the non-source-built SDK was working is that we were accidentally still using DependencyModel 2.2.
Further, we can't really rely on the xunit fix as we need to be able to run dotnet test on projects using older xunit with latest SDK. The JSON parsing code in xunit is pulled in from a package reference.
So we need to change DependencyModel to not escape forward slahshes.
cc @ericstj