Coverage percentage is wrong
tlevesque-ueat opened this issue · 6 comments
Bug Report
I think an image summarizes the problem well enough:
I'm not sure how 0.8587... was rounded to 0.87, it should be 0.86
Then it gets really confusing, because the value is displayed as 87%, which is the threshold value, but it claims that the line coverage is below the threshold (which is technically true, but not if you consider the value that is displayed)
Workflow File
- name: Code Coverage Summary Report
uses: irongut/CodeCoverageSummary@v1.3.0
with:
filename: [REDACTED]/coverage.cobertura.xml
badge: true
fail_below_min: true
format: markdown
hide_branch_rate: false
hide_complexity: true
indicators: true
output: both
thresholds: '87 77'Actions Log
Run irongut/CodeCoverageSummary@v1.3.0
with:
filename: [REDACTED]/coverage.cobertura.xml
badge: true
fail_below_min: true
format: markdown
hide_branch_rate: false
hide_complexity: true
indicators: true
output: both
thresholds: 87 77
env:
DOTNET_ROOT: /home/runner/.dotnet
/usr/bin/docker run --name ghcrioirongutcodecoveragesummaryv130_454770 --label 94859b --workdir /github/workspace --rm -e DOTNET_ROOT -e INPUT_FILENAME -e INPUT_BADGE -e INPUT_FAIL_BELOW_MIN -e INPUT_FORMAT -e INPUT_HIDE_BRANCH_RATE -e INPUT_HIDE_COMPLEXITY -e INPUT_INDICATORS -e INPUT_OUTPUT -e INPUT_THRESHOLDS -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_RUN_ATTEMPT -e GITHUB_ACTOR -e GITHUB_TRIGGERING_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_REF_NAME -e GITHUB_REF_PROTECTED -e GITHUB_REF_TYPE -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e GITHUB_STEP_SUMMARY -e RUNNER_OS -e RUNNER_ARCH -e RUNNER_NAME -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e ACTIONS_ID_TOKEN_REQUEST_URL -e ACTIONS_ID_TOKEN_REQUEST_TOKEN -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/[REDACTED]":"/github/workspace" ghcr.io/irongut/codecoveragesummary:v1.3.0 "--files" "[REDACTED]/coverage.cobertura.xml,[REDACTED]/coverage.cobertura.xml" "--badge" "true" "--fail" "true" "--format" "markdown" "--hidebranch" "false" "--hidecomplexity" "true" "--indicators" "true" "--output" "both" "--thresholds" "87 77"
Coverage File: /github/workspace/[REDACTED]/coverage.cobertura.xml
Coverage File: /github/workspace/[REDACTED]/coverage.cobertura.xml
0Coverage-87%25-critical?style=flat)
Package | Line Rate | Branch Rate | Health
-------- | --------- | ----------- | ------
[REDACTED] | 89% | 67% | ➖
[REDACTED] | 84% | 70% | ❌
**Summary** | **87%** (1021 / 1189) | **68%** (17[3](https://github.com/[REDACTED]?check_suite_focus=true#step:11:3) / 2[5](https://github.com/[REDACTED]?check_suite_focus=true#step:11:5)0) | ❌
_Minimum allowed line rate is `8[7](https://github.com/[REDACTED]?check_suite_focus=true#step:11:7)%`_
FAIL: Overall line rate below minimum threshold of [8](https://github.com/[REDACTED]?check_suite_focus=true#step:11:8)7%.
Expected Behavior
The line coverage should be reported at 86%, not 87%
Additional Context
N/A
Linked To
N/A
The branch coverage is also wrong btw: it says 68%, but 173/250 is 0.692, so 69%...
@tlevesque-ueat Is this reading a single coverage file or two coverage files? If it's a single file then those values come from the coverage file with no additional maths on my part. It it is two files then I have a problem.
It's a little difficult without access to your coverage file since most tools output at least 4 decimal places but from the output shown...
I'm not sure how 0.8587... was rounded to 0.87, it should be 0.86
(0.89 + 0.84) / 2 = 0.865, rounded to 87%
This also explains why the coverage failed but it is confusing given the value displayed.
The branch coverage is also wrong btw: it says 68%, but 173/250 is 0.692, so 69%...
(0.67 + 0.70) / 2 = 0.685, since this was rounded down the package values must be displayed rounded up
This would probably be less confusing (and more accurate) if I didn't round the percentages to whole numbers. Based on sample coverage files, 2 decimal places would probably be better.
Hi @irongut,
Is this reading a single coverage file or two coverage files? If it's a single file then those values come from the coverage file with no additional maths on my part. It it is two files then I have a problem.
It's taking two files as input.
It's a little difficult without access to your coverage file since most tools output at least 4 decimal places but from the output shown...
For confidentiality reasons, I don't think I can provide the files... sorry. The line rate in the XML files has 4 decimal places (but I don't have the files for this exact run...)
@irongut - thank you for your work, one thing I noticed was the way you are calculating combined report coverage. You can't average two percentages that math doesn't work, you have to add the lines covered and uncovered from each project and use those to recalculate the percentage.
(0.89 + 0.84) / 2 = 0.865, rounded to 87%
@tlevesque-ueat - issue doesn't really show you the severity of this miscalculation but let's say you have this scenario
| pkg | covered lines | total lines | % covered |
|---|---|---|---|
| package 1 | 90 | 100 | 90% |
| package 2 | 70 | 1000 | 7% |
based on your previous comment and the behavior I have seen using the glob pattern it looks like you are calculating the combined percentage with this formula
(0.90 + 0.07) / 2 = 0.485
but the actual calculation should be
(90 + 70) / (100 + 1000) = .145
As you can see this is a much bigger issue
My current workaround is to have all of my output using the json format to a common directory and then using nyc report --temp-dir='./coverage/gathered-reports' --reporter=cobertura to do the combining into a single output report and not using the glob pattern
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days.
This issue was closed because it has been stale for 30 days with no activity.