Coverage Output File Needs to exist?
agray opened this issue · 18 comments
I have this in pipeline.yml file:
- name: Run Unit Tests
uses: b3b00/coverlet-action@1.1.1
with:
testProject: ${{ env.UT_PROJECTDIR }}
output: 'opencover.info'
outputFormat: 'opencover'
excludes: '[program]*,[UnitTests]Tests.*,[Merlin]*'
env.UT_PROJECTDIR is defined as:
UT_PROJECTDIR: UnitTests\\ # unit test directory
The output I get is:
Run b3b00/coverlet-action@1.1.1
create coverlet args
found exclusions [program],[UnitTests]Tests.,[Merlin]*
coverlet args -p:coverletOutput=opencover.info -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:Exclude="[program],[UnitTests]Tests.,[Merlin]*" UnitTests\
run dotnet test
Determining projects to restore...
All projects are up-to-date for restore.
Saucery -> /home/runner/work/Saucery/Saucery/Saucery/bin/Debug/net5.0/Saucery.dll
UnitTests -> /home/runner/work/Saucery/Saucery/UnitTests/bin/Debug/net5.0/UnitTests.dll
Test run for /home/runner/work/Saucery/Saucery/UnitTests/bin/Debug/net5.0/UnitTests.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.11.0
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Skipped FlowControlTest [< 1 ms]
Passed! - Failed: 0, Passed: 41, Skipped: 1, Total: 42, Duration: 829 ms - /home/runner/work/Saucery/Saucery/UnitTests/bin/Debug/net5.0/UnitTests.dll (net5.0)
Error: error occured : coverage file not found at ./opencover.info
[1]covergae file not found at ./opencover.info
setting output coverageFile : >>./opencover.info<<
coverlet test done.
I interpret the error message as requiring the output file to exist !? I would have expected this output file to be WRITTEN by the action and it SHOULDN'T exist.
This doesn't seem right. What am I missing? Must be something silly on my part.
Thanks in advance.
hello @agray , output file is not required to exist. I think the problem is witn the testprohect arg of the action. it needs to be the path to the unit test csproj file. in your case it only contains the directory name.
it should be somethin like
-testproject: UnitTests/UnitTests.csproj
hopes this will help.
Unfortunately it still fails:
- name: Run Unit Tests
uses: b3b00/coverlet-action@1.1.1
with:
testProject: ${{ env.UT_PROJECT }}
output: 'opencover.info'
outputFormat: 'opencover'
excludes: '[program]*,[UnitTests]Tests.*,[Merlin]*'
env.UT_PROJECT is defined as:
UT_PROJECT: UnitTests/UnitTests.csproj # unit test directory
And my output is:
Run b3b00/coverlet-action@1.1.1
create coverlet args
found exclusions [program],[UnitTests]Tests.,[Merlin]*
coverlet args -p:coverletOutput=opencover.info -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:Exclude="[program],[UnitTests]Tests.,[Merlin]*" UnitTests/UnitTests.csproj
run dotnet test
Determining projects to restore...
All projects are up-to-date for restore.
Saucery -> /home/runner/work/Saucery/Saucery/Saucery/bin/Debug/net5.0/Saucery.dll
UnitTests -> /home/runner/work/Saucery/Saucery/UnitTests/bin/Debug/net5.0/UnitTests.dll
Test run for /home/runner/work/Saucery/Saucery/UnitTests/bin/Debug/net5.0/UnitTests.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.11.0
Copyright (c) Microsoft Corporation. All rights reserved.Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Skipped FlowControlTest [< 1 ms]Passed! - Failed: 0, Passed: 41, Skipped: 1, Total: 42, Duration: 818 ms - /home/runner/work/Saucery/Saucery/UnitTests/bin/Debug/net5.0/UnitTests.dll (net5.0)
Error: error occured : coverage file not found at UnitTests/opencover.info
[1]covergae file not found at UnitTests/opencover.info
setting output coverageFile : >>UnitTests/opencover.info<<
coverlet test done.
Can we please have this reopened.
@agray i' ve reopened the issue. could you point me to you repo so i can look closer at your workflow
@agray I've published a new release of coverlet-action (1.1.2.2) with debug information. Could you try it (I've tried on a fork but I miss some secrets)
Finally igot my own saucage api user and keys.
And I've found your issue. You miss a coverlet package reference in the unittests.csproj
You can check it at this commit : https://github.com/b3b00/Saucery/commit/72838716ed658a735f1fa2b3aeff0982218ee536
And matching pipeline run : https://github.com/b3b00/Saucery/actions/runs/1277666968
Thank you @b3b00
Here is my run (still in progress at time of writing):
https://github.com/Sauceforge/Saucery/runs/3718732765?check_suite_focus=true
Not sure why it is taking 8 minutes and counting on the unit test step though. Hmm.
Looks like it waiting for a hosted runner:
2021-09-27T09:43:46.5932871Z Found online and idle hosted runner in the current repository's organization account that matches the required labels: 'ubuntu-latest'
2021-09-27T09:43:46.7026437Z Waiting for a Hosted runner in the 'organization' to pick this job...
2021-09-27T09:43:47.1011290Z Job is waiting for a hosted runner to come online.
2021-09-27T09:43:51.3109154Z Job is about to start running on the hosted runner: Hosted Agent (hosted)
Cancelled and reran build. Worked that time.
https://github.com/Sauceforge/Saucery/runs/3718847744?check_suite_focus=true
Thank you @b3b00, I was clearly the PEBCAK ;)
Question: Does the action support failing the build if the coverage is too low? Can that be configured in the pipeline yml?
i ve just added a threshold param to set a minimal coverage. i ve not test it though you can have a try with last release. check readme for documentation.
Have look at this:
https://github.com/Sauceforge/Saucery/runs/3718980448?check_suite_focus=true
Specifically:
treshold, should be threshold I think.
Is this what I'm supposed to see?
https://github.com/Sauceforge/Saucery/runs/3719548987?check_suite_focus=true
Error message is not really explicit but lowering threshold to 10 make the build pass.
I've used the default coverlet threshold mechnism and error message seems to "disappear". For now it's working but I will later look at a way to make a clear error message or return a output variable stating that there is a coverage threshold error.
Thank you @b3b00 for your assistance.
The exception threw me (no pun intended). Was expecting a "clean" error message to indicate build failure.
https://github.com/Sauceforge/Saucery/runs/3719660453?check_suite_focus=true (I need more unit tests :/)
I guess that the issue comes from github actions runner that does not display standard error stream. coverlet display an error message like this :
error : The minimum line coverage is below the specified 80
on standard error output so we don't see it :( I am trying to find a way to redirect stderr to stdout. If I can not do it I will look at a way to manage myself the threshold not relying on coverlet own mechanism