Test compatibility with other tools
irongut opened this issue · 11 comments
irongut commented
Feature Request
CCS has been tested with Coverlet and gcov, test with other tools that output Cobertura format.
Expected Behaviour
Obtain sample Coberura output from as many testing tools as possible, CCS should parse them correctly.
Additional Context
Linked To
#20 What does complexity mean? (issue with coverage.py)
#22 Allow hiding some output values
irongut commented
coverage.py Sample
<coverage branch-rate="0" branches-covered="0" branches-valid="0" complexity="0" line-rate="0.6500" lines-covered="65" lines-valid="100" timestamp="1635484070186" version="6.0.2">
<packages>
<package branch-rate="0" complexity="0" line-rate="0.6500" name="one" />
<package branch-rate="0" complexity="0" line-rate="0.6000" name="two" />
<package branch-rate="0" complexity="0" line-rate="0.7000" name="three" />
</packages>
</coverage>(Details not relevant to CCS omitted.)
pboling commented
Tried it with simplecov & simplecov-cobertura, and it failed, but that may be caused by the options I have enabled not being supported by simplecov? If that's the case this error should be handled more gracefully or avoided altogether.
Here's the log:
Run irongut/CodeCoverageSummary@v1.[2](https://github.com/oauth-xx/oauth2/runs/5226726966?check_suite_focus=true#step:9:2).0
with:
filename: ./coverage/coverage.xml
badge: true
fail_below_min: true
format: markdown
hide_branch_rate: false
hide_complexity: true
indicators: true
output: both
thresholds: 95 97
env:
CC_TEST_REPORTER_ID: ***
GIT_COMMIT_SHA: 5e7798ac8bfdc4f[3](https://github.com/oauth-xx/oauth2/runs/5226726966?check_suite_focus=true#step:9:3)d8537d[4](https://github.com/oauth-xx/oauth2/runs/5226726966?check_suite_focus=true#step:9:4)23af23be[5](https://github.com/oauth-xx/oauth2/runs/5226726966?check_suite_focus=true#step:9:5)f33ec[6](https://github.com/oauth-xx/oauth2/runs/5226726966?check_suite_focus=true#step:9:6)81
GIT_BRANCH: 1-4-stable
/usr/bin/docker run --name ghcrioirongutcodecoveragesummaryv120_db1[7](https://github.com/oauth-xx/oauth2/runs/5226726966?check_suite_focus=true#step:9:7)64 --label 7db2d7 --workdir /github/workspace --rm -e CC_TEST_REPORTER_ID -e GIT_COMMIT_SHA -e GIT_BRANCH -e INPUT_FILENAME -e INPUT_BADGE -e INPUT_FAIL_BELOW_MIN -e INPUT_FORMAT -e INPUT_HIDE_BRANCH_RATE -e INPUT_HIDE_COMPLEXITY -e INPUT_INDICATORS -e INPUT_OUTPUT -e INPUT_THRESHOLDS -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_RUN_ATTEMPT -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_REF_NAME -e GITHUB_REF_PROTECTED -e GITHUB_REF_TYPE -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_ARCH -e RUNNER_NAME -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/oauth2/oauth2":"/github/workspace" ghcr.io/irongut/codecoveragesummary:v1.2.0 "--files" "./coverage/coverage.xml" "--badge" "true" "--fail" "true" "--format" "markdown" "--hidebranch" "false" "--hidecomplexity" "true" "--indicators" "true" "--output" "both" "--thresholds" "[9](https://github.com/oauth-xx/oauth2/runs/5226726966?check_suite_focus=true#step:9:9)5 97"
Code Coverage File: ./coverage/coverage.xml
Parse Error: Sequence contains no elements
Error: System.NullReferenceException - Object reference not set to an instance of an object.
Going to tweak options and see if I can get it working.
pboling commented
Simplecov Sample (Ruby, branch coverage not enabled)
<?xml version='1.0'?>
<!DOCTYPE coverage SYSTEM "http://cobertura.sourceforge.net/xml/coverage-04.dtd">
<!-- Generated by simplecov-cobertura version 2.1.0 (https://github.com/dashingrocket/simplecov-cobertura) -->
<coverage line-rate="0.97" lines-covered="350" lines-valid="362" complexity="0" version="0" timestamp="1645060052">
<sources>
<source>/Users/pboling/src/my/oauth2</source>
</sources>
<packages>
<package name="oauth2" line-rate="0.97" complexity="0">
<classes>
<class name="oauth2" filename="lib/oauth2.rb" line-rate="1.0" complexity="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="2" hits="1"/>
<line number="3" hits="1"/>
<line number="4" hits="1"/>
<line number="5" hits="1"/>
<line number="6" hits="1"/>
<line number="7" hits="1"/>
<line number="8" hits="1"/>
<line number="9" hits="1"/>
<line number="10" hits="1"/>
<line number="11" hits="1"/>
<line number="12" hits="1"/>
</lines>
</class>
<class name="access_token" filename="lib/oauth2/access_token.rb" line-rate="0.96" complexity="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="2" hits="1"/>
<line number="3" hits="1"/>
<line number="4" hits="1"/>
<line number="7" hits="1"/>
<line number="13" hits="1"/>
<line number="14" hits="7"/>
<line number="15" hits="7"/>
<line number="23" hits="1"/>
<line number="24" hits="1"/>
<line number="41" hits="1"/>
<line number="42" hits="151"/>
<line number="43" hits="151"/>
<line number="44" hits="151"/>
<line number="45" hits="151"/>
<line number="46" hits="453"/>
<line number="48" hits="151"/>
<line number="49" hits="151"/>
<line number="50" hits="151"/>
<line number="51" hits="151"/>
<line number="52" hits="151"/>
<line number="55" hits="151"/>
<line number="61" hits="1"/>
<line number="62" hits="6"/>
<line number="68" hits="1"/>
<line number="69" hits="11"/>
<line number="75" hits="1"/>
<line number="76" hits="3"/>
<line number="83" hits="1"/>
<line number="84" hits="2"/>
<line number="86" hits="2"/>
<line number="87" hits="2"/>
<line number="88" hits="2"/>
<line number="89" hits="2"/>
<line number="90" hits="2"/>
<line number="91" hits="2"/>
<line number="97" hits="1"/>
<line number="98" hits="1"/>
<line number="107" hits="1"/>
<line number="108" hits="20"/>
<line number="109" hits="20"/>
<line number="115" hits="1"/>
<line number="116" hits="2"/>
<line number="122" hits="1"/>
<line number="123" hits="18"/>
<line number="129" hits="1"/>
<line number="130" hits="2"/>
<line number="136" hits="1"/>
<line number="137" hits="0"/>
<line number="143" hits="1"/>
<line number="144" hits="2"/>
<line number="148" hits="1"/>
<line number="149" hits="8"/>
<line number="152" hits="1"/>
<line number="154" hits="1"/>
<line number="155" hits="20"/>
<line number="157" hits="8"/>
<line number="158" hits="8"/>
<line number="160" hits="8"/>
<line number="161" hits="8"/>
<line number="163" hits="4"/>
<line number="164" hits="4"/>
<line number="165" hits="4"/>
<line number="167" hits="0"/>
<line number="171" hits="0"/>
<line number="175" hits="1"/>
<line number="176" hits="13"/>
<line number="178" hits="12"/>
</lines>
</class>
<class name="authenticator" filename="lib/oauth2/authenticator.rb" line-rate="1.0" complexity="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="3" hits="1"/>
<line number="4" hits="1"/>
<line number="5" hits="1"/>
<line number="7" hits="1"/>
<line number="8" hits="106"/>
<line number="9" hits="106"/>
<line number="10" hits="106"/>
<line number="22" hits="1"/>
<line number="23" hits="106"/>
<line number="25" hits="14"/>
<line number="27" hits="89"/>
<line number="29" hits="1"/>
<line number="31" hits="1"/>
<line number="33" hits="1"/>
<line number="37" hits="1"/>
<line number="38" hits="15"/>
<line number="41" hits="1"/>
<line number="45" hits="1"/>
<line number="46" hits="89"/>
<line number="51" hits="1"/>
<line number="52" hits="1"/>
<line number="57" hits="1"/>
<line number="58" hits="14"/>
<line number="59" hits="14"/>
<line number="60" hits="14"/>
<line number="64" hits="1"/>
<line number="65" hits="14"/>
</lines>
</class>
<class name="client" filename="lib/oauth2/client.rb" line-rate="0.95" complexity="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="2" hits="1"/>
<line number="4" hits="1"/>
<line number="6" hits="1"/>
<line number="7" hits="1"/>
<line number="9" hits="1"/>
<line number="10" hits="1"/>
<line number="11" hits="1"/>
<line number="31" hits="1"/>
<line number="32" hits="204"/>
<line number="33" hits="204"/>
<line number="34" hits="204"/>
<line number="35" hits="204"/>
<line number="36" hits="204"/>
<line number="39" hits="204"/>
<line number="49" hits="204"/>
<line number="55" hits="1"/>
<line number="56" hits="0"/>
<line number="57" hits="0"/>
<line number="61" hits="1"/>
<line number="62" hits="423"/>
<line number="63" hits="158"/>
<line number="64" hits="158"/>
<line number="65" hits="134"/>
<line number="66" hits="134"/>
<line number="69" hits="158"/>
<line number="76" hits="1"/>
<line number="77" hits="11"/>
<line number="78" hits="11"/>
<line number="84" hits="1"/>
<line number="85" hits="100"/>
<line number="100" hits="1"/>
<line number="101" hits="141"/>
<line number="103" hits="141"/>
<line number="105" hits="141"/>
<line number="106" hits="141"/>
<line number="107" hits="141"/>
<line number="109" hits="141"/>
<line number="111" hits="141"/>
<line number="113" hits="3"/>
<line number="114" hits="3"/>
<line number="115" hits="3"/>
<line number="117" hits="2"/>
<line number="118" hits="1"/>
<line number="119" hits="1"/>
<line number="121" hits="2"/>
<line number="124" hits="127"/>
<line number="126" hits="11"/>
<line number="127" hits="11"/>
<line number="129" hits="2"/>
<line number="130" hits="2"/>
<line number="132" hits="0"/>
<line number="133" hits="0"/>
<line number="143" hits="1"/>
<line number="144" hits="97"/>
<line number="145" hits="182"/>
<line number="146" hits="2"/>
<line number="148" hits="180"/>
<line number="151" hits="97"/>
<line number="153" hits="97"/>
<line number="154" hits="97"/>
<line number="155" hits="97"/>
<line number="156" hits="97"/>
<line number="157" hits="79"/>
<line number="158" hits="79"/>
<line number="160" hits="18"/>
<line number="161" hits="18"/>
<line number="163" hits="97"/>
<line number="164" hits="97"/>
<line number="167" hits="97"/>
<line number="169" hits="0"/>
<line number="172" hits="97"/>
<line number="173" hits="3"/>
<line number="174" hits="3"/>
<line number="176" hits="94"/>
<line number="182" hits="1"/>
<line number="183" hits="48"/>
<line number="189" hits="1"/>
<line number="190" hits="5"/>
<line number="196" hits="1"/>
<line number="197" hits="11"/>
<line number="203" hits="1"/>
<line number="204" hits="21"/>
<line number="207" hits="1"/>
<line number="208" hits="9"/>
<line number="227" hits="1"/>
<line number="228" hits="55"/>
<line number="229" hits="2"/>
<line number="231" hits="53"/>
<line number="235" hits="1"/>
<line number="236" hits="91"/>
<line number="237" hits="91"/>
<line number="240" hits="1"/>
<line number="242" hits="1"/>
<line number="243" hits="97"/>
<line number="244" hits="97"/>
<line number="246" hits="93"/>
<line number="250" hits="93"/>
<line number="251" hits="1"/>
<line number="253" hits="92"/>
</lines>
</class>
<class name="error" filename="lib/oauth2/error.rb" line-rate="1.0" complexity="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="2" hits="1"/>
<line number="3" hits="1"/>
<line number="7" hits="1"/>
<line number="8" hits="14"/>
<line number="9" hits="14"/>
<line number="11" hits="14"/>
<line number="12" hits="7"/>
<line number="13" hits="7"/>
<line number="14" hits="7"/>
<line number="17" hits="14"/>
<line number="23" hits="1"/>
<line number="24" hits="14"/>
<line number="26" hits="14"/>
<line number="28" hits="14"/>
<line number="29" hits="7"/>
<line number="30" hits="7"/>
<line number="32" hits="7"/>
<line number="35" hits="14"/>
<line number="37" hits="14"/>
</lines>
</class>
<class name="mac_token" filename="lib/oauth2/mac_token.rb" line-rate="0.95" complexity="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="2" hits="1"/>
<line number="3" hits="1"/>
<line number="4" hits="1"/>
<line number="6" hits="1"/>
<line number="7" hits="1"/>
<line number="14" hits="1"/>
<line number="15" hits="3"/>
<line number="18" hits="1"/>
<line number="30" hits="1"/>
<line number="31" hits="19"/>
<line number="32" hits="19"/>
<line number="34" hits="18"/>
<line number="43" hits="1"/>
<line number="44" hits="4"/>
<line number="46" hits="4"/>
<line number="47" hits="4"/>
<line number="49" hits="4"/>
<line number="53" hits="1"/>
<line number="54" hits="1"/>
<line number="61" hits="1"/>
<line number="62" hits="9"/>
<line number="63" hits="9"/>
<line number="65" hits="9"/>
<line number="67" hits="8"/>
<line number="69" hits="7"/>
<line number="71" hits="7"/>
<line number="80" hits="1"/>
<line number="82" hits="8"/>
<line number="91" hits="8"/>
<line number="97" hits="1"/>
<line number="98" hits="19"/>
<line number="101" hits="1"/>
<line number="103" hits="0"/>
<line number="107" hits="17"/>
<line number="109" hits="0"/>
<line number="112" hits="1"/>
<line number="116" hits="1"/>
<line number="120" hits="1"/>
<line number="124" hits="1"/>
<line number="125" hits="8"/>
</lines>
</class>
<class name="response" filename="lib/oauth2/response.rb" line-rate="1.0" complexity="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="2" hits="1"/>
<line number="3" hits="1"/>
<line number="5" hits="1"/>
<line number="7" hits="1"/>
<line number="8" hits="1"/>
<line number="9" hits="1"/>
<line number="14" hits="62"/>
<line number="15" hits="44"/>
<line number="16" hits="3"/>
<line number="20" hits="1"/>
<line number="32" hits="1"/>
<line number="33" hits="3"/>
<line number="34" hits="3"/>
<line number="35" hits="3"/>
<line number="36" hits="6"/>
<line number="46" hits="1"/>
<line number="47" hits="147"/>
<line number="48" hits="147"/>
<line number="52" hits="1"/>
<line number="53" hits="6"/>
<line number="57" hits="1"/>
<line number="58" hits="150"/>
<line number="62" hits="1"/>
<line number="63" hits="154"/>
<line number="69" hits="1"/>
<line number="70" hits="134"/>
<line number="72" hits="131"/>
<line number="76" hits="1"/>
<line number="77" hits="244"/>
<line number="81" hits="1"/>
<line number="82" hits="244"/>
<line number="84" hits="244"/>
<line number="89" hits="1"/>
<line number="90" hits="1"/>
</lines>
</class>
<class name="assertion" filename="lib/oauth2/strategy/assertion.rb" line-rate="0.89" complexity="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="3" hits="1"/>
<line number="4" hits="1"/>
<line number="23" hits="1"/>
<line number="27" hits="1"/>
<line number="28" hits="1"/>
<line number="45" hits="1"/>
<line number="46" hits="8"/>
<line number="47" hits="8"/>
<line number="50" hits="1"/>
<line number="51" hits="8"/>
<line number="53" hits="8"/>
<line number="60" hits="1"/>
<line number="62" hits="8"/>
<line number="67" hits="8"/>
<line number="68" hits="8"/>
<line number="69" hits="0"/>
<line number="70" hits="0"/>
</lines>
</class>
<class name="auth_code" filename="lib/oauth2/strategy/auth_code.rb" line-rate="1.0" complexity="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="2" hits="1"/>
<line number="6" hits="1"/>
<line number="10" hits="1"/>
<line number="11" hits="3"/>
<line number="17" hits="1"/>
<line number="18" hits="3"/>
<line number="27" hits="1"/>
<line number="28" hits="44"/>
<line number="30" hits="44"/>
</lines>
</class>
<class name="base" filename="lib/oauth2/strategy/base.rb" line-rate="1.0" complexity="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="2" hits="1"/>
<line number="3" hits="1"/>
<line number="4" hits="1"/>
<line number="5" hits="95"/>
</lines>
</class>
<class name="client_credentials" filename="lib/oauth2/strategy/client_credentials.rb" line-rate="1.0" complexity="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="2" hits="1"/>
<line number="6" hits="1"/>
<line number="10" hits="1"/>
<line number="11" hits="1"/>
<line number="18" hits="1"/>
<line number="19" hits="20"/>
<line number="20" hits="20"/>
</lines>
</class>
<class name="implicit" filename="lib/oauth2/strategy/implicit.rb" line-rate="1.0" complexity="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="2" hits="1"/>
<line number="6" hits="1"/>
<line number="10" hits="1"/>
<line number="11" hits="3"/>
<line number="17" hits="1"/>
<line number="18" hits="3"/>
<line number="24" hits="1"/>
<line number="25" hits="1"/>
</lines>
</class>
<class name="password" filename="lib/oauth2/strategy/password.rb" line-rate="1.0" complexity="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="2" hits="1"/>
<line number="6" hits="1"/>
<line number="10" hits="1"/>
<line number="11" hits="1"/>
<line number="19" hits="1"/>
<line number="20" hits="10"/>
<line number="23" hits="10"/>
</lines>
</class>
</classes>
</package>
</packages>
</coverage>
irongut commented
Hi @pboling thanks for the sample, that's always useful.
Looking at the sample there are no branch metrics on coverage or package nodes. Unfortunately the hide_branch_rate option only hides it from the report, the action still attempts to calculate branch metrics.
It's funny how many different ways different tools can output what is supposed to be the same format. I'll need to make some changes to prevent attempting to calculate metrics that don't exist in the data.
pboling commented
@irongut Branch coverage can be turned on, but I've never tried it. I'll see what that generates now. But, yeah, definitely good to support the lack of any specific feature. Always assume the null case :)
irongut commented
@pboling Looking at the docs, enable branch coverage but leave primary coverage set to the default of line. That should be compatible with the current version of CCS.
I'll add a note to the CCS docs and include a fix for missing branch rate in the next version.
github-actions commented
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days.
github-actions commented
This issue was closed because it has been stale for 30 days with no activity.
pboling commented
NOTE: the suggested config is working. I wrote up a how to on dev.to.
github-actions commented
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days.
github-actions commented
This issue was closed because it has been stale for 30 days with no activity.