coverlet
Coverlet is a cross platform code coverage library for .NET Core, with support for line, branch and method coverage.
Installation
Global Tool:
dotnet tool install --global coverlet.console
Package Reference:
dotnet add package coverlet.msbuild
How It Works
Coverlet generates code coverage information by going through the following process:
Before Tests Run
- Locates the unit test assembly and selects all the referenced assemblies that have PDBs.
- Instruments the selected assemblies by inserting code to record sequence point hits to a temporary file.
After Tests Run
- Restore the original non-instrumented assembly files.
- Read the recorded hits information from the temporary file.
- Generate the coverage result from the hits information and write it to a file.
Note: The assembly you'd like to get coverage for must be different from the assembly that contains the tests
Usage
Coverlet can be used either as a .NET Core global tool that can be invoked from a terminal or as a NuGet package that integrates with the MSBuild system of your test project.
Global Tool
To see a list of options, run:
coverlet --help
The current options are (output of coverlet --help
):
Cross platform .NET Core code coverage tool 1.0.0.0
Usage: coverlet [arguments] [options]
Arguments:
<ASSEMBLY> Path to the test assembly.
Options:
-h|--help Show help information
-v|--version Show version information
-t|--target Path to the test runner application.
-a|--targetargs Arguments to be passed to the test runner.
-o|--output Output of the generated coverage report
-f|--format Format of the generated coverage report.
--threshold Exits with error if the coverage % is below value.
--threshold-type Coverage type to apply the threshold to.
--threshold-stat Coverage statistic used to enforce the threshold value.
--exclude Filter expressions to exclude specific modules and types.
--include Filter expressions to include specific modules and types.
--include-directory Include directories containing additional assemblies to be instrumented.
--exclude-by-file Glob patterns specifying source files to exclude.
--include-directory Include directories containing additional assemblies to be instrumented.
--exclude-by-attribute Attributes to exclude from code coverage.
--merge-with Path to existing coverage result to merge.
--use-source-link Specifies whether to use SourceLink URIs in place of file system paths.
Code Coverage
The coverlet
tool is invoked by specifying the path to the assembly that contains the unit tests. You also need to specify the test runner and the arguments to pass to the test runner using the --target
and --targetargs
options respectively. The invocation of the test runner with the supplied arguments must not involve a recompilation of the unit test assembly or no coverage data will be generated.
The following example shows how to use the familiar dotnet test
toolchain:
coverlet /path/to/test-assembly.dll --target "dotnet" --targetargs "test /path/to/test-project --no-build"
After the above command is run, a coverage.json
file containing the results will be generated in the directory the coverlet
command was run. A summary of the results will also be displayed in the terminal.
Note: The --no-build
flag is specified so that the /path/to/test-assembly.dll
isn't rebuilt
Coverage Output
Coverlet can generate coverage results in multiple formats, which is specified using the --format
or -f
options. For example, the following command emits coverage results in the opencover
format instead of json
:
coverlet <ASSEMBLY> --target <TARGET> --targetargs <TARGETARGS> --format opencover
Supported Formats:
- json (default)
- lcov
- opencover
- cobertura
- teamcity
The --format
option can be specified multiple times to output multiple formats in a single run:
coverlet <ASSEMBLY> --target <TARGET> --targetargs <TARGETARGS> --format opencover --format lcov
By default, Coverlet will output the coverage results file(s) in the current working directory. The --output
or -o
options can be used to override this behaviour.
coverlet <ASSEMBLY> --target <TARGET> --targetargs <TARGETARGS> --output "/custom/path/result.json"
The above command will write the results to the supplied path, if no file extension is specified it'll use the standard extension of the selected output format. To specify a directory instead, simply append a /
to the end of the value.
coverlet <ASSEMBLY> --target <TARGET> --targetargs <TARGETARGS> --output "/custom/directory/" -f json -f lcov
TeamCity Output
Coverlet can output basic code coverage statistics using TeamCity service messages.
coverlet <ASSEMBLY> --target <TARGET> --targetargs <TARGETARGS> --output teamcity
The currently supported TeamCity statistics are:
TeamCity Statistic Key | Description |
---|---|
CodeCoverageL | Line-level code coverage |
CodeCoverageC | Class-level code coverage |
CodeCoverageM | Method-level code coverage |
CodeCoverageAbsLTotal | The total number of lines |
CodeCoverageAbsLCovered | The number of covered lines |
CodeCoverageAbsCTotal | The total number of classes |
CodeCoverageAbsCCovered | The number of covered classes |
CodeCoverageAbsMTotal | The total number of methods |
CodeCoverageAbsMCovered | The number of covered methods |
Merging Results
With Coverlet you can combine the output of multiple coverage runs into a single result.
coverlet <ASSEMBLY> --target <TARGET> --targetargs <TARGETARGS> --merge-with "/path/to/result.json" --format opencover
The value given to --merge-with
must be a path to Coverlet's own json result format.
Threshold
Coverlet allows you to specify a coverage threshold below which it returns a non-zero exit code. This allows you to enforce a minimum coverage percent on all changes to your project.
coverlet <ASSEMBLY> --target <TARGET> --targetargs <TARGETARGS> --threshold 80
The above command will automatically fail the build if the line, branch or method coverage of any of the instrumented modules falls below 80%. You can specify what type of coverage to apply the threshold value to using the --threshold-type
option. For example to apply the threshold check to only line coverage:
coverlet <ASSEMBLY> --target <TARGET> --targetargs <TARGETARGS> --threshold 80 --threshold-type line
You can specify the --threshold-type
option multiple times. Valid values include line
, branch
and method
.
coverlet <ASSEMBLY> --target <TARGET> --targetargs <TARGETARGS> --threshold 80 --threshold-type line --threshold-type method
By default, Coverlet will validate the threshold value against the coverage result of each module. The --threshold-stat
option allows you to change this behaviour and can have any of the following values:
- Minimum (Default): Ensures the coverage result of each module isn't less than the threshold
- Total: Ensures the total combined coverage result of all modules isn't less than the threshold
- Average: Ensures the average coverage result of all modules isn't less than the threshold
The following command will compare the threshold value with the overall total coverage of all modules:
coverlet <ASSEMBLY> --target <TARGET> --targetargs <TARGETARGS> --threshold 80 --threshold-type line --threshold-stat total
Excluding From Coverage
Attributes
You can ignore a method or an entire class from code coverage by creating and applying the ExcludeFromCodeCoverage
attribute present in the System.Diagnostics.CodeAnalysis
namespace.
You can also ignore additional attributes by using the ExcludeByAttribute
property (short name or full name supported):
coverlet <ASSEMBLY> --target <TARGET> --targetargs <TARGETARGS> --exclude-by-attribute "Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute"
Source Files
You can also ignore specific source files from code coverage using the --exclude-by-file
option
- Can be specified multiple times
- Use absolute or relative paths (relative to the project directory)
- Use file path or directory path with globbing (e.g
dir1/*.cs
)
coverlet <ASSEMBLY> --target <TARGET> --targetargs <TARGETARGS> --exclude-by-file "../dir1/class1.cs"
Filters
Coverlet gives the ability to have fine grained control over what gets excluded using "filter expressions".
Syntax: --exclude '[Assembly-Filter]Type-Filter'
Wildcards
*
=> matches zero or more characters?
=> the prefixed character is optional
Examples
--exclude "[*]*"
=> Excludes all types in all assemblies (nothing is instrumented)--exclude "[coverlet.*]Coverlet.Core.Coverage"
=> Excludes the Coverage class in theCoverlet.Core
namespace belonging to any assembly that matchescoverlet.*
(e.gcoverlet.core
)--exclude "[*]Coverlet.Core.Instrumentation.*"
=> Excludes all types belonging toCoverlet.Core.Instrumentation
namespace in any assembly--exclude "[coverlet.*.tests?]*"
=> Excludes all types in any assembly starting withcoverlet.
and ending with.test
or.tests
(the?
makes thes
optional)--exclude "[coverlet.*]*" --exclude "[*]Coverlet.Core*"
=> Excludes assemblies matchingcoverlet.*
and excludes all types belonging to theCoverlet.Core
namespace in any assembly
coverlet <ASSEMBLY> --target <TARGET> --targetargs <TARGETARGS> --exclude "[coverlet.*]Coverlet.Core.Coverage"
Coverlet goes a step in the other direction by also letting you explicitly set what can be included using the --include
option.
Examples
--include "[*]*"
=> Includes all types in all assemblies (everything is instrumented)--include "[coverlet.*]Coverlet.Core.Coverage"
=> Includes the Coverage class in theCoverlet.Core
namespace belonging to any assembly that matchescoverlet.*
(e.gcoverlet.core
)--include "[coverlet.*.tests?]*"
=> Includes all types in any assembly starting withcoverlet.
and ending with.test
or.tests
(the?
makes thes
optional)
Both --exclude
and --include
options can be used together but --exclude
takes precedence. You can specify the --exclude
and --include
options multiple times to allow for multiple filter expressions.
MSBuild
In this mode, Coverlet doesn't require any additional setup other than including the NuGet package in the unit test project. It integrates with the dotnet test
infrastructure built into the .NET Core CLI and when enabled, will automatically generate coverage results after tests are run.
If a property takes multiple comma-separated values please note that you will have to add escaped quotes around the string like this: /p:Exclude=\"[coverlet.*]*,[*]Coverlet.Core*\"
, /p:Include=\"[coverlet.*]*,[*]Coverlet.Core*\"
, or /p:CoverletOutputFormat=\"json,opencover\"
.
Note for Powershell / VSTS users
To exclude or include multiple assemblies when using Powershell scripts or creating a .yaml file for a VSTS build %2c
should be used as a separator. Msbuild will translate this symbol to ,
.
/p:Exclude="[*]*Examples?%2c[*]*Startup"
VSTS builds do not require double quotes to be unescaped:
dotnet test --configuration $(buildConfiguration) --no-build /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Build.SourcesDirectory)/TestResults/Coverage/ /p:Exclude="[MyAppName.DebugHost]*%2c[MyAppNamet.WebHost]*%2c[MyAppName.App]*"
Code Coverage
Enabling code coverage is as simple as setting the CollectCoverage
property to true
dotnet test /p:CollectCoverage=true
After the above command is run, a coverage.json
file containing the results will be generated in the root directory of the test project. A summary of the results will also be displayed in the terminal.
Coverage Output
Coverlet can generate coverage results in multiple formats, which is specified using the CoverletOutputFormat
property. For example, the following command emits coverage results in the opencover
format:
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=opencover
Supported Formats:
- json (default)
- lcov
- opencover
- cobertura
- teamcity
You can specify multiple output formats by separating them with a comma (,
).
The output of the coverage result can be specified using the CoverletOutput
property.
dotnet test /p:CollectCoverage=true /p:CoverletOutput='./result.json'
To specify a directory where all results will be written to (especially if using multiple formats), end the value with a /
.
dotnet test /p:CollectCoverage=true /p:CoverletOutput='./results/'
Merging Results
With Coverlet you can combine the output of multiple coverage runs into a single result.
dotnet test /p:CollectCoverage=true /p:MergeWith='/path/to/result.json'
The value given to /p:MergeWith
must be a path to Coverlet's own json result format.
Threshold
Coverlet allows you to specify a coverage threshold below which it fails the build. This allows you to enforce a minimum coverage percent on all changes to your project.
dotnet test /p:CollectCoverage=true /p:Threshold=80
The above command will automatically fail the build if the line, branch or method coverage of any of the instrumented modules falls below 80%. You can specify what type of coverage to apply the threshold value to using the ThresholdType
property. For example to apply the threshold check to only line coverage:
dotnet test /p:CollectCoverage=true /p:Threshold=80 /p:ThresholdType=line
You can specify multiple values for ThresholdType
by separating them with commas. Valid values include line
, branch
and method
.
By default, Coverlet will validate the threshold value against the coverage result of each module. The /p:ThresholdStat
option allows you to change this behaviour and can have any of the following values:
- Minimum (Default): Ensures the coverage result of each module isn't less than the threshold
- Total: Ensures the total combined coverage result of all modules isn't less than the threshold
- Average: Ensures the average coverage result of all modules isn't less than the threshold
The following command will compare the threshold value with the overall total coverage of all modules:
dotnet test /p:CollectCoverage=true /p:Threshold=80 /p:ThresholdType=line /p:ThresholdStat=total
Excluding From Coverage
Attributes
You can ignore a method or an entire class from code coverage by creating and applying the ExcludeFromCodeCoverage
attribute present in the System.Diagnostics.CodeAnalysis
namespace.
You can also ignore additional attributes by using the ExcludeByAttribute
property (short name or full name supported):
dotnet test /p:CollectCoverage=true /p:ExcludeByAttribute="Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute"
Source Files
You can also ignore specific source files from code coverage using the ExcludeByFile
property
- Use single or multiple paths (separate by comma)
- Use absolute or relative paths (relative to the project directory)
- Use file path or directory path with globbing (e.g
dir1/*.cs
)
dotnet test /p:CollectCoverage=true /p:ExcludeByFile=\"../dir1/class1.cs,../dir2/*.cs,../dir3/**/*.cs,\"
Filters
Coverlet gives the ability to have fine grained control over what gets excluded using "filter expressions".
Syntax: /p:Exclude=[Assembly-Filter]Type-Filter
Wildcards
*
=> matches zero or more characters?
=> the prefixed character is optional
Examples
/p:Exclude="[*]*"
=> Excludes all types in all assemblies (nothing is instrumented)/p:Exclude="[coverlet.*]Coverlet.Core.Coverage"
=> Excludes the Coverage class in theCoverlet.Core
namespace belonging to any assembly that matchescoverlet.*
(e.gcoverlet.core
)/p:Exclude="[*]Coverlet.Core.Instrumentation.*"
=> Excludes all types belonging toCoverlet.Core.Instrumentation
namespace in any assembly/p:Exclude="[coverlet.*.tests?]*"
=> Excludes all types in any assembly starting withcoverlet.
and ending with.test
or.tests
(the?
makes thes
optional)/p:Exclude=\"[coverlet.*]*,[*]Coverlet.Core*\"
=> Excludes assemblies matchingcoverlet.*
and excludes all types belonging to theCoverlet.Core
namespace in any assembly
dotnet test /p:CollectCoverage=true /p:Exclude="[coverlet.*]Coverlet.Core.Coverage"
Coverlet goes a step in the other direction by also letting you explicitly set what can be included using the Include
property.
Examples
/p:Include="[*]*"
=> Includes all types in all assemblies (everything is instrumented)/p:Include="[coverlet.*]Coverlet.Core.Coverage"
=> Includes the Coverage class in theCoverlet.Core
namespace belonging to any assembly that matchescoverlet.*
(e.gcoverlet.core
)/p:Include="[coverlet.*.tests?]*"
=> Includes all types in any assembly starting withcoverlet.
and ending with.test
or.tests
(the?
makes thes
optional)
Both Exclude
and Include
properties can be used together but Exclude
takes precedence.
You can specify multiple filter expressions by separting them with a comma (,
).
SourceLink
Coverlet supports SourceLink custom debug information contained in PDBs. When you specify the --use-source-link
flag in the global tool or /p:UseSourceLink=true
property in the MSBuild command, Coverlet will generate results that contain the URL to the source files in your source control instead of absolute file paths.
Cake Addin
If you're using Cake Build for your build script you can use the Cake.Coverlet addin to provide you extensions to dotnet test for passing coverlet arguments in a strongly typed manner.
Roadmap
- Merging outputs (multiple test projects, one coverage result)
- Support for more output formats (e.g. JaCoCo)
- Console runner (removes the need for requiring a NuGet package)
Issues & Contributions
If you find a bug or have a feature request, please report them at this repository's issues section. Contributions are highly welcome, however, except for very small changes, kindly file an issue and let's have a discussion before you open a pull request.
Building The Project
Clone this repo:
git clone https://github.com/tonerdo/coverlet
Change directory to repo root:
cd coverlet
Execute build script:
dotnet msbuild build.proj
This will result in the following:
- Restore all NuGet packages required for building
- Build and publish all projects. Final binaries are placed into
<repo_root>\build\<Configuration>
- Build and run tests
These steps must be followed before you attempt to open the solution in an IDE (e.g. Visual Studio, Rider) for all projects to be loaded successfully.
Performance testing
There is a simple performance test for the hit counting instrumentation in the test project coverlet.core.performancetest
. Build the project with the msbuild step above and then run:
dotnet test /p:CollectCoverage=true test/coverlet.core.performancetest/
The duration of the test can be tweaked by changing the number of iterations in the [InlineData]
in the PerformanceTest
class.
For more realistic testing it is recommended to try out any changes to the hit counting code paths on large, realistic projects. If you don't have any handy https://github.com/dotnet/corefx is an excellent candidate. This page describes how to run code coverage tests for both the full solution and for individual projects with coverlet from nuget. Suitable projects (listed in order of escalating test durations):
- System.Collections.Concurrent.Tests
- System.Collections.Tests
- System.Reflection.Metadata.Tests
- System.Xml.Linq.Events.Tests
- System.Runtime.Serialization.Formatters.Tests
Change to the directory of the library and run the msbuild code coverage command:
dotnet msbuild /t:BuildAndTest /p:Coverage=true
To run with a development version of coverlet call dotnet run
instead of the installed coverlet version, e.g.:
dotnet msbuild /t:BuildAndTest /p:Coverage=true /p:CoverageExecutablePath="dotnet run -p C:\...\coverlet\src\coverlet.console\coverlet.console.csproj"
Code of Conduct
This project enforces a code of conduct in line with the contributor covenant. See CODE OF CONDUCT for details.
License
This project is licensed under the MIT license. See the LICENSE file for more info.