Version 4.7.1221 creates no coverage compared to version 4.7.922
Lorilatschki opened this issue · 20 comments
NOTE Support requests should initially be raised on GitHub discussions
My Framework
- .NET 2
- .NET 3.5
- .NET 4
- .NET 4.5
- .NET 4.6
- .NET 4.6.1
- .NET 4.6.2
- .NET 4.7
- .NET 4.7.1
- .NET 4.7.2
- .NET 4.8
- .NET Core 2.2.0
- .NET Core 3.1.0
- .NET 5
- only release and LTS versions of .NET runtimes and SDKs will be supported
My Environment
- Windows 7 or below (not truly supported due to EOL)
- Windows 8
- Windows 8.1
- Windows 10
- Windows 10 IoT Core
- Windows Server 2012
- Windows Server 2012 R2
- Windows Server 2016
- Windows Server 2019
I have already...
- repeated the problem using the latest stable release of OpenCover.
- reviewed the usage guide and usage document.
- have looked at the opencover output xml file in an attempt to resolve the issue.
- reviewed the current issues to check that the issue isn't already known.
My issue is related to (check only those which apply):
- no coverage being recorded
- 64 bit support
Describe the bug
We try to switch from version 4.7.922 to 4.7.1221. With the version 4.7.922 everything works as expected, line and branches are covered as expected. When I switch to the new version 4.7.1221 some assemblies getting zero coverage! The only thing which has changed between two runs is the OpenCover version, nothing else.
We are using the following arguments:
-register:path64-filter:xxx-coverbytest:*.Test*.dll-returntargetcode:0-threshold:10
Even when I remove all those arguments, no coverage is being created with version 4.7.1221!
Tests run: 464, Errors: 0, Failures: 0, Inconclusive: 0, Time: 11,6616257 seconds
Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0
Committing...
Visited Classes 0 of 4321 (0)
Visited Methods 0 of 36457 (0)
Visited Points 0 of 149799 (0)
Visited Branches 0 of 99897 (0)
==== Alternative Results (includes all methods including those without corresponding source) ====
Alternative Visited Classes 0 of 4459 (0)
Alternative Visited Methods 0 of 40190 (0)
We are running our tests with NUnit 2.6.2 build against .Net Framework v4.0.
I have also tried version 4.7.1189 but it also behaves like 4.7.1221 and covers nothing.
Steps to reproduce the problem:
- reviewed the usage guide and usage document again because we know you probably didn't do it the first time.
- I am sorry, but I cannot share any code/assembly since it is confidential.
I ran into the same problem.
v4.7.922 is good, but v4.7.1189 is not ==> All my nunit tests are executed, but none visited.
In my case, I have 3 test DLLs, one of which gets coverage stats on both issues, while the other 2 do not on newer versions.
The difference between the two: the ones that stopped working load other mixed-mode assemblies that in turn load native DLLs.
I can confirm this problem - I've discovered this only because the tool Fine Code Coverage uses 4.7.922, while I tried to use the most recent one, which is 4.7.1221.
I'm experiencing this with the vstest console runner on .NET Framework 4.5.2.
@Lorilatschki @sjmla @wowanator are you able to provide a sample project of it failing for you so I can replicate?
@sawilde sorry for the late response. Unfortunately I am not able to provide such an sample project since the sources/assemblies are inner source. I can try to create one, but maybe @sjmla or @wowanator are able to share there projects in between.
I tried to create a sample project, but, of course, cannot replicate the problem.
So, either I am missing something, or I was wrong about mixed-mode assemblies having anything to do with it...
I'll attach the project just in case anyone else has better luck or want to pick up on this as a starting point...
If you can only repeat locally then you may need to debug this yourself - I can't fix something I am unable to replicate.
I would start by using DebugView to see any runtime errors emitted from the profiler, increase logging verbosity to full using the -log:All switch.
So, weirdly enough, after upgrading my old school project to sdk-style, and updating from VS2017 to VS2019, I can upgrade to the latest OpenCover and generate coverage reports without any issues.
@Lorilatschki, perhaps you can try that...?
@sjmla that is a little freaky - can you provide more detail of the changes - perhaps the devil is in the detail.
Not much in terms of details.
Ran
dotnet tool install --global Project2015To2017.Migrate2019.Tool, followed by
dotnet migrate-2019 wizard my.sln
Fixed breakages, and moved from using the VS2017-bundled MSBuild to use the VS2019 Build-Tools's MSBuild. That's it...
So, weirdly enough, after upgrading my old school project to sdk-style, and updating from VS2017 to VS2019, I can upgrade to the latest OpenCover and generate coverage reports without any issues.
@Lorilatschki, perhaps you can try that...?
Sorry for the late response. But currently we are not able to migrate to VS2019 and the sdk style format.
@sawilde is there a way I can debug this locally to identify the issue? Since you know your code best, can you point me to the code part's I need to add break points to in order to find out, why the profiler does not catch any lines of code for some of our assemblies?
Since you know you code best,
I wish, most of this code was written 5+ years ago and I have not looked it at much in the past 3 years except for a few minor changes. However, I'll give it a go.
@Lorilatschki I would start by using the debugview tool and see if anything useful pops out of there.
My suspicion is the PDB format is no longer supported by the library I am using to read them - the PDB format kept changing (or so it appeared) with each version of .net and .net core - making managing OpenCover a bit of a 'mare. It's stabalised since which is why I suspect upgrading to the latest and greatest resolved the issue for others.
So I would look here (assuming you can build/debug OpenCover) - https://github.com/OpenCover/opencover/blob/master/main/OpenCover.Framework/Symbols/CecilSymbolManager.cs#L123
I don't have a windows machine anymore and so I am unable to support this project much more until I get a replacement.
@sawilde I had some time to analyse the issue. Here is what I currently found out:
Regarding build
after installing all required software packages I was not able to build the solution out of the box. I had following two compile errors:
- The opencover.snk file does not exist in the version folder! Is it generated somehow? I solved this locally by duplicating the opencover.test.snk file and rename it to opencover.snk.
- The profiler project also does not build from the scratch. It requires a version.h file which in fact also does not exist in the version folder. I solved this locally by removing the version.h include and setting the desired versions directly inside the OpenCover.Profiler.rc file.
Regarding the issue itself
I need some hints/advices, because it turned out, that no exception is thrown when covering the
bad caseassembly and attaching a debugger to it. It looks like the assembly which is references by the test is not loaded by the memory profiler at all and thus nothing is collected. Therefor I added some console outputs to MessageHandler.cs. In the good case, the assemblies which are referenced by the test are loaded properly.
The only difference between the good and the bad case are, that in the bad case, the assembly to test refers a c++ dll.
Is there a way to get a bit deeper inside the memory profiling mechanism, in order to find out why no StandardMessage with type MSG_Type.MSG_TrackAssembly for the bad case assembly is fired!?
both 1 and 2 are generated if you run the build from the command line - the CI service has a stashed copy of the official snk file (this acts as type of official signing)
Is it a pure c++ assembly or one designed to run on the .net runtime? Only .net assemblies are loaded by the runtime and fed to the profiler.
Is it a pure c++ assembly or one designed to run on the .net runtime? Only .net assemblies are loaded by the runtime and fed to the profiler.
No, it is a .net assembly build against net48 x64. I have double checked it, it even has no reference to a unmanagged c++ assembly at all (I thought this was the difference but I was wrong). Is there a way to fetch exceptions of the c++ x64 profiler with the hope to catch the one indicating why some of our productive assemblies aren't covered anymore with version 4.7.922?
The crazy stuff is, that all assemblies are build inside the same pipeline/workflow, but only some of them aren't covered within the latest opencover version.
you can try using debugview and turning up the verbosity - if you have a debug build it'll be pretty noisy but may give you clues. Usually though if the runtime loads your assembly then it should be recorded by the opencover profiler.
IIRC when an assembly is loaded the call should come through here
You can always uncomment the ATLTRACE statement to get more info in debug builds
There seem to be a failure on profiler site. I have masked the assembly names...
...
[37440] OpenCover: (Profiler) ::ModuleAttachedToAssembly(...) => (8FC52370 => D:\bin\goodAssemblyA.dll, 1EB3C390 => goodAssemblyA)
[37440] OpenCover: (Profiler) ::ModuleAttachedToAssembly(...) => (8FE398F8 => D:\bin\goodAssemblyB.dll, 1F55EF50 => goodAssemblyB)
[37440] OpenCover: (Profiler) ::ModuleAttachedToAssembly(...) => (8FE3BB20 => D:\bin\goodAssemblyC.dll, 1F55F2B0 => goodAssemblyC)
[37440] OpenCover: (Profiler) ::ThreadAssignedToOSThread(532682544, 31492)
[37440] OpenCover: (Profiler) ProfilerCommunication::RequestInformation(...) => Communication (Chat channel - TrackAssembly) with host has failed (0x102, 10000)
...
I will try to add a native c++ debugger (or mixed mode) to check if I can find the reason why the RequestInformation fails for the mentioned assembly. The last time I was programming/debugging c++ stuff was about 10 years ago 🤣. I give it a try 😇
I found the reason why this happened. 😛
For some reason the track assembly is getting a timeout for that special assembly (Default ist 10000). After increasing the timeout to 60000 the assembly got profiled as expected.
I have no glue why this takes so long for that one assembly, however it might be a solution to use the environment variable OpenCover_Profiler_CommWait to increase the timeout during our coverage run!
After double checking that inside our test pipeline and ensuring that this solves the issue, I will close it!
Many thanks for you support @sawilde so far! You helped me alot!
I believe that is manageable from the command line eg
-communicationtimeout:<integer, e.g. 10000>
I believe that is manageable from the command line eg
-communicationtimeout:<integer, e.g. 10000>
Yes this is even better to use the command line cli instead of the env variable. After changing to a timeout of 3 minutes, it seems to work with the latest package.
Have you any idea why the profiler takes that long to instrument some of our assemblies in comparison to the version 4.7.922?
Some work was done on excluding the hidden branches points that people get so upset about - I am surprised the impact would be that much though but then I don't have a .net project of large enough size to test it against anymore.