.NET Core Tool Support
StrangeWill opened this issue · 69 comments
Any plans to support the new "dotnet" command? Considering dnx command support is dropped and the new tooling should allow for better self-configuration over the old DNX method found here: https://github.com/OpenCover/opencover/wiki/DNX-Support
Currently not yet - the runtime does not fully support the Profiling API and to also fully support the platform would require us to build the profiler virtually from scratch to run on linux platform, something I do not have the bandwidth for. If you build your libraries as PCL I believe you can also run them on .NET 4.5 etc on windows and get coverage from OpenCover assuming your tooling allows that.
I only develop/enhance OpenCover for my needs and I currently do not have the need; It doesn't mean someone else can't do the work but it probably won't be me in the near future.
Hi @StrangeWill,
I've managed to get it running using the following command :
opencover.console.exe -register:user -target:"C:/Program Files/dotnet/dotnet.exe" -targetargs:"test test/MyProject.Test" -output:"coverage.xml"
Make sure you have xunit-runner on your test project. Please let me know if you encounter any problems.
@haythem Awesome! Thanks I'll try that out later this week.
I tried
c:\Users\tmds\AppData\Local\Apps\OpenCover\OpenCover.Console.exe "-target:C:\Program Files\dotnet\dotnet.exe" -targetargs:test -register:user -filter:"+[*]* -[xunit*]*" -output:coverage.xml
This generates a report, but the report shows zero coverage for all the code.
What could be wrong?
I had this running with dotnet RC2 too, but when upgrading to final it stopped working on my CI machine (Windows 2012 R2), but still works on developer machines (Windows 7).
Any ideas anyone?
What does it say in the XML file for the reason why an assembly of interest was skipped?
Nothing. The assembly was not skipped, you just get an entry with line / branch coverage 0.
Interestingly enough it works when using the oldStyle switch. I also installed VS 2015 Update 3 on our CI server but still the same issue exists.
dotCover is also having the same issue since 1.0.0 came out, wonder if the change that broke them both is related...
might be. only strange thing is that it's only occuring on W2k12R2 (at least that's where it's happening for me). I'll check W10 tonight.
Interestingly enough it works when using the oldStyle switch.
Ah that explains it - OpenCover needs to hook into a class in mscorlib to work around a security issue with pinvoke - the class in question in .NET core is no longer mscorlib so OpenCover can't hook it anymore - The -oldstyle switch does not use that class but you may encounter security issues especially around 'AssemblyAllowsPartialTrust...'
see #595
and it also happens on W10.
and it also happens on W10.
it's not an OS issue it's what they've done to the structure - since there is a workaround I have not made it high on my agenda.
There are two aspects to this issue:
- make it work on .NET Core
- integrate with .NET Core
From what I read here, if you pass -oldstyle it should work.
The second aspect is supporting OpenCover as a dotnet tool. This means you can add it in the project.json tools section, pull it as a dependency and invoke it using the dotnet cli (like Microsoft.EntityFrameworkCore.Tools).
Yeah @tmds I was originally asking about the second aspect, though I'm not sure how "OK" it is to build tools that only work on specific platforms.
make it work on .NET Core
currently you can use the -oldstyle flag until we implment #595
integrate with .NET Core
OpenCover currently only works with the Full .NET FW on windows due to its dependency on the Profiler API. The profiler is written in native c++ as you cannot write the profiler in .NET (unlike Java profilers I believe)
I understand there is some progress in migrating this API to the .NET Core but I am not sure how ready it is see #549. The migration on windows will be relatively easy as the profiler will still use the windows API. The same profiler capability on linux based platforms may take some time.
The second aspect is supporting OpenCover as a dotnet tool.
I am not sure where to start here - we do have nuget support and just added chocolatey support for the next release #603, we can accomodate other packaging formats but I want to be able to automate that as part of the pipeline - e.g. no manual actions after the initial 'kick-off'. But, the tool would still be limited to windows OSs for the time being.
For anyone who wants to use OpenCover on Windows with MSBuild and wants the results in Visual Studio Team Services, we use it like this:
https://gist.github.com/tverboon/1c187b082b9597abef89dcec3df2422c
The conversion to Cobertura is to get it in Visual Studio Team Services.
In addition to the info already provided in this thread: Please note that you can't use
"debugType": "portable" in your project.json build options, otherwise OpenCover fails with pdb's missing.
So another thingie for OpenCover would be the support for the new pdb format.
So another thingie for OpenCover would be the support for the new pdb format.
We get that support from the Mono.Cecil project - @jp7677 could you raise that as a separate issue please that way when we start to resolve it you will be kept informed. preferably with a sample showing how to repeat the issue etc if possible.
I'm sorry for the late response. In fact, I had the same problem after upgrading to AspNetCore 1.0 RTM.
This how I managed to solve it and don't ask me how 😄
c:\Users\tmds\AppData\Local\Apps\OpenCover\OpenCover.Console.exe "-target:C:\Program Files\dotnet\dotnet.exe" -targetargs:test -register:user -filter:"+[*]* -[xunit*]*" -output:coverage.xml -oldStyle
Please let me know if you encounter any other problems.
There are a number of items covered in this issue.
- One can be worked around using the 'oldstyle' switch and will be fixed by #595
- One is related to the profiler api not being available on Linux, see https://github.com/dotnet/coreclr/issues/445
- One is a feature request to be able to use OpenCover as a cli tool (project.json tools section)
It's not clear from the coreclr#445 ticket when profiler api is expected for linux. It looks like there work being done then it went quiet. Was there any discussion on it being 1.0.1 or 1.1.0?
In case it's helpful, you can see a working example of OpenCover running on a .NET Core project here, but I had to use one of the .NET Platform Standard framework designations (i.e. netstandard1.*) for my source project to get it all to work. That enabled running my .NET Core project on Windows (read: Appveyor) with the traditional .NET Framework so I could run OpenCover. While this is unnecessarily restrictive, it isn't crippling since the .NET Platform Standard frameworks enable the broadest portability.
I wrote up the gotchas on this StackOverflow answer here.
I got this working with my .NET core 1.0.0 project. OpenCover version 4.6.519.
I had to change the debug type in my project.json from portable to full:
debugType: "full"
Run the following from the console where OpenCover.Console.exe lives:
OpenCover.Console.exe
"-target:C:\Program Files\dotnet\dotnet.exe"
-targetargs:" test ""path\to\project.json"" "
-register:user
-filter:"+[service_]_ -[service]*Test"
-output:coverage.xml
-oldStyle
The above will generate a coverage.xml file. I then downloaded ReportGenerator and ran the following command to generate an html file based off the coverage.xml file:
ReportGenerator.exe -reports:"path\to\coverage.xml" -targetdir:"path\to\output\directory"
Any there any downsides to using -oldStyle if you do not care about Silverlight? The usage notes are pretty clear it's a last resort kind of thing, but they don't explain the tradeoff.
Any there any downsides to using -oldStyle if you do not care about Silverlight?
It's not just Silverlight - there are some security issues as well that may cause the instrumentation to fail - try it and see - your mileage may vary.
@sawilde does that mean that as long as we do not get failures, the results will be accurate?
Yup - should be.
Can't get OpenCover to work on a multitarget project.
It works fine when running the -f net451. But the same command for -f netcoreapp1.0 is not getting any results but 0 in all values.
Committing...
Visited Classes 0 of 2 (0)
Visited Methods 0 of 3 (0)
Visited Points 0 of 9 (0)
Visited Branches 0 of 3 (0)
==== Alternative Results (includes all methods including those without corresponding source) ====
Alternative Visited Classes 0 of 2 (0)
Alternative Visited Methods 0 of 4 (0)
Here's the command line
D:>"C:\Program Files (x86)\OpenCover\OpenCover.Console.exe" -target:"C:\Program Files\dotnet\dotnet.exe" -targetargs:" test ""d:\data\projects\2015\multipleframeworks\mylibrary\src\mylibrary.tests\project.json"" -f netcoreapp1.0" -register:user -filter:"+[] -[xunit*]*" -returntargetcode -output:"d:\op_results.xml"
D:>dotnet --info
.NET Command Line Tools (1.0.0-preview2-003133)
Product Information:
Version: 1.0.0-preview2-003133
Commit SHA-1 hash: 74df06500c
Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x64
Anybody else having this issue?
@gmartinezsan You have to add the -oldStyle flag for .Net Core.
@tverboon tried and it worked fine. But then opencover installation some how got broken. I'm using WIndows 10 and can't find a way to get it working again. I'm using the msi.
Here's the output
"C:\Program Files (x86)\OpenCover\OpenCover.Console.exe" -target:"C:\Program Files\dotnet\dotnet.exe" -targetargs:" test ""d:\data\projects\2015\multipleframeworks\mylibrary\src\mylibrary.tests\project.json"" -f netcoreapp1.0" -register:user -filter:"+[] -[xunit*]*" -returntargetcode -output:"d:\op_results.xml"
Executing: C:\Program Files\dotnet\dotnet.exe
Unknown option: test
Unknown option: d:\data\projects\2015\multipleframeworks\mylibrary\src\mylibrary.tests\project.json
Unknown option: -f
Unknown option: netcoreapp1.0
.NET Command Line Tools (1.0.0-preview2-003133)
Usage: dotnet [host-options] [command] [arguments] [common-options]
Arguments:
[command] The command to execute
[arguments] Arguments to pass to the command
[host-options] Options specific to dotnet (host)
[common-options] Options common to all commands
Common options:
-v|--verbose Enable verbose output
-h|--help Show help
Host options (passed before the command):
-v|--verbose Enable verbose output
--version Display .NET CLI Version Number
--info Display .NET CLI Info
Common Commands:
new Initialize a basic .NET project
restore Restore dependencies specified in the .NET project
build Builds a .NET project
publish Publishes a .NET project for deployment (including the runtime)
run Compiles and immediately executes a .NET project
test Runs unit tests using the test runner specified in the project
pack Creates a NuGet package
Committing...
No results, this could be for a number of reasons. The most common reasons are:
1) missing PDBs for the assemblies that match the filter please review the
output file and refer to the Usage guide (Usage.rtf) about filters.
2) the profiler may not be registered correctly, please refer to the Usage
guide and the -register switch.
Is this a known issue?
I can't see anything obvious it looks like the arguments being passed to dotnet.exe are no longer recognised.
If you have trouble put your dotnet test ... statement in a batch file and then get opencover to just run the batch file it saves a lot of effort with complex argument passing.
@sawilde thanks. Do you have any example?
@gmartinezsan is your debugType in your project.json set to full instead of portable?
@carusology is set up to full.
"buildOptions": {
"debugType": "full"
},
I understand that OpenCover will work on windows using the oldStyle flag. Do we know if there has been any progression on the instrumentation required to run it on Linux or MacOS? I'm pretty surprised (at Microsoft) that currently there's no documented way for getting code coverage on these OS' for .NET Core since many CI server stacks and providers use them.
The profiler API uses COM and though I have seen an attempt to support the API in .NET on Linux it is not yet ready and I haven't had time to review its status - I don't currently use .NET core in my day to day work and hence I have no real drive to develop it at this time.
@gmartinezsan Thank you for your comment!
When using "dotnet migrate" to port my projects from "project.json" type builds to ".csproj" type builds, it seems the default behavior has changed:
I specified no debugType in my project.json and OpenCover succeeded by default.
I specified no debugType in my .csproj and OpenCover failed by default.
Adding the node
<DebugType>full</DebugType>
to all my .csproj files fixed my issue.
One additional note: do not forget to configure <DebugType>full</DebugType> or Project Properties>Build>Advnaced for all core projects (not only test)
Core projects by default generate portable PDB that are not supported by opencover
@thepadawan @rpokrovskij
I added the <DebugType>full</DebugType> to all csproj files and added the /p:DebugType=full to target args but I get zero coverage, what am I missing? (using vs 2017 dotnet 1.0.2)
the following is my command line:
C:\Users\user\.nuget\packages\opencover\4.6.519\tools\OpenCover.Console.exe -register:user -target:"C:\Program Files\dotnet\dotnet.exe" -targetargs:"test /p:DebugType=full D:\Github\IsraelHikingMap\Site\Tests\IsraelHiking.API.Tests\IsraelHiking.API.Tests.csproj" -filter:"+[*]*API* +[*]*Database* +[*]*GPSBabel* -[*]*JsonResponse* -[*]*GpxTypes* -[*]*Tests*" -excludebyattribute:"*.ExcludeFromCodeCoverage*" -output:D:\Github\IsraelHikingMap\Site\coverage-opencover.xml
console results:
Committing...
Visited Classes 0 of 144 (0)
Visited Methods 0 of 435 (0)
Visited Points 0 of 2559 (0)
Visited Branches 0 of 1010 (0)
==== Alternative Results (includes all methods including those without corresponding source) ====
Alternative Visited Classes 0 of 144 (0)
Alternative Visited Methods 0 of 654 (0)
@HarelM
But where is -oldStyle ? And simplify filters till all inclusive -filter: "+[*]*".
I have summarized my experience there: https://github.com/rpokrovskij/opencover4vs.ps1
This powershell script generates OpenCover input parameters both for Core and Classic frameworks (xUnit aka dotnet -test and NUnit aka mstest) accepting test projects path globbing as parameter (*.Test.csproj).
@HarelM I get different results of "no coverage" from you:
Committing... No results, this could be for a number of reasons. The most common reasons are: 1) missing PDBs for the assemblies that match the filter please review the output file and refer to the Usage guide (Usage.rtf) about filters. 2) the profiler may not be registered correctly, please refer to the Usage guide and the -register switch.
Meaning "OpenCover could not find any coverage results" as opposed to your error message, "OpenCover found coverage results, and they are 0."
I suggest you try @rpokrovskij 's suggestion or try loosening your filters until you get non-zero coverage results.
I didn't got these results before using the <DebugType>full</DebugType>.
I'll try the -oldStlye as soon as I get my laptop back from repair.
Thank you both for the quick response!
@thepadawan Seems that -oldstyle flag does solve the issue of coverage.
So summarize my steps:
- Add
<DebugType>full</DebugType>to all the projects (probably not needed for all, but just to make sure) - Run the following command line:
C:\Users\user\.nuget\packages\opencover\4.6.519\tools\OpenCover.Console.exe -register:user -target:"C:\Program Files\dotnet\dotnet.exe" -targetargs:"test --logger:trx;LogFileName=results.trx /p:DebugType=full full-path-to-text-project.csproj" -filter:"+[*]*" -output:full-path-to-results-file.xml -oldstyle
@HarelM @thepadawan You could try to use pdbonly if this is still a valid option for .Net Core. To do coverage you need the pdbs. For development DebugType Full is fine, but there used to be a condition for Configuration Release to set DebugType pdbonly in standard csproj files. You could also try to override it with commandline flag again.
@HarelM Great news, thank you for the full command line!
@tverboon Thank you for the idea! Right now, this is not a priority for me, as I build all my projects using the configuration Debug and only use them for internal development. When I take the step towards releasing production-level nuget packages, I will have to take steps to reduce the DebugType, and will come back to this thread with any issues.
@tverboon @thepadawan @HarelM I have started work on the .net core support
The library I am dependant on is due to release soon and so I can start my integration using the latest beta - it also supports portable pdb and embedded as well so hopefully a lot of these workarounds you have found will disappear soon.
@sawilde Thanks for the update, keep me posted so I can remove the workaround in the future although I don't have a lot of dlls so the debugtype doesn't bother me much.
@tverboon .Net core configuration is different from regular .Net, I'm familiar with the ability to change pdb in regular .Net projects and you can have configuration dependent flag in .Net core but it's not exactly the same as it used to be, in any case I was able to run a .Net core project in AppVeyor with coverage for both .Net and javascript code using powershell, if anyone is interested the code can be found here:
https://github.com/IsraelHikingMap/Site/blob/master/AppVeyorBuild/PostBuildTests.ps1#L70
@HarelM @thepadawan @rpokrovskij Thank you all!!.
Its work for me after converting .Net core project from vs2015 to vs2017. Added <DebugType>full</DebugType> to required solution files.
@kalpeshsatasiya I have a beta build (not in nuget yet though) that should handle portable
https://ci.appveyor.com/project/sawilde/opencover/build/4.6.690
Any updates on an expected timeline for this? Nothing I'd try to hold anyone to, but it would be nice to have a rough idea, partly in terms of whether I should start looking for alternative coverage solutions.
I have a dot net core project which is running CI with coverage using open cover.
Results can be seen here:
https://coveralls.io/github/IsraelHikingMap/Site
CI build can be seen here (last few build have an issue with NuGet which is an appvoyor problem not with the code):
https://ci.appveyor.com/project/HarelM/site
Project sources can be found here:
https://github.com/IsraelHikingMap/Site
@HarelM: That's using debugType=full though, which is a problem in my case - I was hoping for the portable pdb support mentioned earlier.
You can completely ignore my comment then... :-)
@jskeet We have got around this limitation by having our CI pipeline copy the project file, transforming the debugType to full from portable, and running code coverage on the copied / transformed variant. Is that tenable for your case? If not, I'm actually quite curious what you have run into that prevents that from being a solution.
@carusology: That would be feasible, but it sounds like a lot of work to automate (which would have to be done in a "meta" way given that we have a lot of projects...) and it'll slow things down as well, building another copy of everything. I'd rather not do that work if the proper fix is reasonably imminent...
as I have said in the past I am waiting for the new mono.cecil release which is perpetual beta at the mo - I could release with the beta but I am bit wary.
@sawilde not to be the straw that broke the camels back - I really appreciate your massive contribution to the community through this tool - but would you consider releasing an alpha/beta with the mono.cecil beta?
Well, if it helps, our multi-project solutions are organized like so:
/
/src
/Foo.Bar
/Foo.Biz
/Foo.Baz
/test
/Foo.Bar.Tests
/Foo.Biz.Tests
/Foo.Baz.Tests
Then we use this script to find all of our *.csproj files and convert their DebugType from portable to full programmatically.
This has been evolving in the coreclr repo: https://github.com/dotnet/coreclr/issues/445#issuecomment-338014386
Coverage is showing as 0.00 %
I have used beow command
"%LOCALAPPDATA%\Apps\OpenCover\OpenCover.Console.exe" -output:"%CD%\opencover.xml" -oldstyle -target:%NUNITCONSOLE%\NUnit.Console-3.7.0\nunit3-console.exe -targetargs:"E:\ProjectFiles\CODE\GRM.Search"\Web API"\SearchService\SearchService.Test\bin\Debug\SearchService.Test.dll" -register:user
In the output xml file showing Missing PDB
<Module skippedDueTo="MissingPdb" hash="73-2C-23-38-69-24-C6-47-B0-15-58-8D-B3-63-F3-64-AC-79-40-D5">
<ModulePath>E:\SONAR\NUnit.Console-3.7.0\nunit3-console.exe</ModulePath>
<ModuleTime>2017-11-14T07:09:46.6084668Z</ModuleTime>
<ModuleName>nunit3-console</ModuleName>
Any users of https://github.com/rpokrovskij/opencover4vs.ps1 ? Need feedback.
Using XUnit2.2.0 in .Net Core 2.0. Using -oldStyle as suggested but it still complains about missing PDBs (not able to recogniz the PDB). Below is the command running ... any suggestion
C:\Users\UN\.nuget\packages\opencover\4.6.519\tools> .\OpenCover.Console.exe -target:"C:\Program Files\dotnet\dotnet.exe" -targetargs:"test D:\Code_Trials\unit-testing-using-dotnet-
test\PrimeService.Tests\PrimeService.Tests.csproj --no-build /p:DebugType=full" -filter:"+[*]* -[*.Tests*]*" -register:user -output:"_CodeCoverageResult.xml" -oldStyle
@smukherjee-rythmos support for .net core is limited in the current version of OpenCover
@sawilde I am keeping an eye on issue #595. Looks like the current version not able to recognize the PDB's since .Net Core generates portable PDB. Any workaround for this? I believe one was to use oldStyle switch but no luck with that. Not sure how come other mentioned that it worked for them? When we will have full .Net Core support with OpenCover, any timeline?
When we will have full .Net Core support with OpenCover, any timeline?
Nothing specific - too much else going on and too many versions of dotnet core to keep up with - now that mono.cecil has come out of beta I may update it and see if that helps everyone.
FYI, an alternative for coverage reporting on NetCore (also for Linux) would be https://github.com/tonerdo/coverlet
on windows .net core is supported both full, embedded and portable formats - only .net (core) old frameworks and 2.2, 3.1 and 5.0 supported
4.7.1137-rc onwards