Code Coverage Options With Combined Coverage Results

[jhornsb2]

Context

I know it is possible to get code coverage using the custom parameters option like this:

- name: Code Coverage
  uses: game-ci/unity-test-runner@v2
  id: tests
  env:
    UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
  with:
    artifactsPath: code-coverage-artifacts
    githubToken: ${{ secrets.GITHUB_TOKEN }}
    checkName: Code Coverage Test Results
    customParameters: -debugCodeOptimization -enableCodeCoverage -coverageOptions generateHtmlReport;generateBadgeReport;assemblyFilters:+<assets> -coverageResultsPath code-coverage-artifacts

This works great for getting the coverage results for a single editmode or playmode test run, but it is unable to get the combined coverage results from running something similar to the following script:

./Unity.exe -projectPath <project-path> -batchmode -testPlatform editmode -runTests -debugCodeOptimization -enableCodeCoverage -coverageResultsPath <results-path> -coverageOptions "generateAdditionalMetrics;assemblyFilters:+<assets>"

./Unity.exe -projectPath <project-path> -batchmode -testPlatform playmode -runTests -debugCodeOptimization -enableCodeCoverage -coverageResultsPath <results-path> -coverageOptions "generateAdditionalMetrics;assemblyFilters:+<assets>"

./Unity.exe -projectPath "C:/Users/Joshua/Documents/Code Projects/GrizzledOwl/common-ci-cd" -batchmode -debugCodeOptimization -enableCodeCoverage -coverageResultsPath <results-path> -coverageOptions "generateHtmlReport;generateBadgeReport;assemblyFilters:+<assets>" -quit

In a lot of cases this combined report is what a lot of people will want when looking at code coverage.

With the information from this the people/teams using this script would be able to use that metric to enforce a minimum code coverage standard on their repositories.

Suggested solution

In order to support the above, I would suggest adding in a "codeCoverage" option flag to the action. I would have it work as follows:

1. When the "testMode" option is "editmode" - creates the code coverage report for editmode tests.
2. When the "testMode" option is "playmode" - creates the code coverage report for playmode tests.
3. When the "testMode" option is "all" - generate combined coverage result.

I would probably add a flag "coverageOnly" so that the test results could be skipped if they aren't needed.

Not only would this support the feature, but it would make it a little clearer to people who are trying to use this action how to get coverage results. In my opinion, it would also make their GitHub actions a little bit more readable.

I would also add an option for "coverageReportType" that would let you specify which reports you are wanting. This would have three options:

1. html - generates the html coverage report
2. badge - generates the coverage summary badges
3. both - generates both the html report and summary badges

I would have this default to the "both" option.

I would add an option for "assemblyFilters" that would let you define what assemblies to include. This would just place your options into the correct place.

To round this out, I would add "coverageOptions". This would be the direct input for the coverageOptions that will be in the batch mode call. Honestly, with this coverageReportType and assemblyFilters wouldn't be needed, but since those options are the ones that are used most often, I think it makes sense to specify them up front in order to make it easier and more readable for people using the action.

Finally, I would add a "minimumCoverage" option. It wouldn't be required, would default to 0, and the allowed values would be a number between 0 and 100. If this is provided, when the coverage results are in, the script would perform a check to see if the coverage meets or exceeds the provided value. If it does, it passes. If it doesn't, it fails.

Using these options would look something like this:

- name: Code Coverage
  uses: game-ci/unity-test-runner@v2
  id: tests
  env:
    UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
  with:
    artifactsPath: code-coverage-artifacts
    githubToken: ${{ secrets.GITHUB_TOKEN }}
    checkName: Code Coverage Test Results
    testMode: all
    codeCoverage:
    coverageOnly:
    coverageReportType: both
    assemblyFilters: +<assets>
    # coverageOptions: verbosity? generateTestReferences? sourcePaths? etc...
    customParameters: -debugCodeOptimization -burst-disable-compilation
    minimumCoverage: 80

Considered alternatives

I don't really have any alternatives, outside of writing my own custom script, to handle these things.

Additional details

I thought about writing my own custom script to handle these things, but it seems like something others would want as well. I am already using game-ci in my other GitHub actions for my projects, so I wanted to see if this was a feature that would be desired here.

I have seen other code coverage issues and pull requests, but I wasn't sure if they were just dropped because they only covered the basic solution that can be handled with customParameters, or if directly supporting coverage wasn't desired for this repository.

I was planning on forking the repository and implementing this myself, but I wanted to discuss my solution, and see if this was even desired.

[nicholas-maltbie]

I've been looking at the repo and it looks like it could probably be added as a flag to the end of this file, https://github.com/game-ci/unity-test-runner/blob/main/dist/steps/run_tests.sh

Maybe add a new flag like "-enableCodeCoverage"

And at if both "$PLAY_MODE" = "true" and "$EDIT_MODE" = "true" and the code coverage flag is enabled (as a separate clause added at the end), we could add a clause to add custom flags based on if code coverage is included.

I might fork this repo tomorrow to see if I can get it working on my own project for validation. I'll post an update if I come up with anything.

[nicholas-maltbie]

I'm working on drafting a PR for this addition right now, my first time using node in a while so code might be a bit off.

I don't think it will work yet (still need to test it) but working on just setting out the boiler plate code with three parameters

• enableCodeCoverage (optional flag, default of disabled)
• assemblyFilters (optional, requires enableCodeCoverage)
• coverageResultsPath (optional, requires enableCodeCoverage)

https://github.com/nicholas-maltbie/unity-test-runner/tree/nicholas-maltbie/codecoverage-support

I'll work on testing it, but have to get on a flight soon so will probably have some update in the next few days! hopefully can test it with my project soon.

[jhornsb2]

Oh, I was planning on working on the solution for this myself. I was just checking to see if this was a desired change since there have been other issues and pull requests related to code coverage.

[nicholas-maltbie]

Alright, I debugged the changes and finishing up all the additions for these requirements of code coverage. Here is a link to the PR, I included some tests and docs to make sure it works as expected. I'll describe the scenarios that I tried to include (I think I have all the ones you outlined form this issue). If you see any issues or have any suggestions, let me know @jhornsb2 :)

#182

[jhornsb2]

@nicholas-maltbie I just got the chance to look through all of the changes. Thank you for implementing it. It looks great and should cover exactly what I need.

[spencer-cheng]

@nicholas-maltbie, @davidmfinol Thanks for your work on the code coverage. I tried to run game-ci/unity-test-runner@v2 in a self-hosted windows runner. I did not provide coverageOptions parameter so the default value is used. I can see the tests are run successfully and the test artifact result is expected. Also, code coverage folder is generated with content. However, I got 0% coverage for all files. When I run the test from the cmd with the same set of parameters locally I get the expected code coverage result which is not 0% for some files. Do you guys have any idea why this is happening? Thanks.

The image below is the command list captured from game-ci/unity-test-runner@v2 log which is the same as the one I used locally except for the path difference.

[nicholas-maltbie]

Hi @spencer-cheng , i ran into this issue as well. I found that explicitly including the assemblies I want to generate coverage options for such as 'assemblyFilters:+my.assembly.*' may help. Are there coverage files generated in the output folder that or is the result just empty?

[spencer-cheng]

Hi, @nicholas-maltbie Thanks for your reply. I tried to explicit include them. But it does not help.
I did more tests in the past two days. It works when i run it on "ubuntu-latest".
However the same code does not when i choose "windows-2019". Similarly, if i run it using a self-hosted windows 2019 runner, the action does not work. If i change to use "run unity command line directly" in step instead of using unity-test-runner action, the coverage data is generated correctly as expected.
I guess there might be a problem with code coverage's windows implementation. Thanks.

[nicholas-maltbie]

@spencer-cheng , there are a few differences between unity when it runs on windows versus Linux in CLI mode. Are you able to produce coverage files properly when running on your local windows machine via cli directly? It might be good to isolate if it's a windows problem when running with CLI or a problem with the way the test runner is configured.

If you're able, could you share the configuration you're using to generate test coverage? Having this workaround of using the run unity command line directly may prove useful to others if this issue takes a while to resolve for windows containers.

[spencer-cheng]

@nicholas-maltbie I am able to produce coverage files properly when running on my local windows machine via CLI directly.

My workaround is like:

1. install Unity on the Windows self-hosted runner and activate the license
2. instead of using unity-test-runner action, i use a step like

 - name: run tests directly from unity
    shell: pwsh
    run: |
      cd dir/path/to/run_test.bat
      ./run_test.bat ${{ env.unity_exe_path}} ${{env.TARGET_TEST_DIR}} ${{env.TARGET_COVERAGE_DIR}}
     echo "::set-output name=artifactsPath::${{ env.TARGET_TEST_DIR}}"
     echo "::set-output name=coveragePath::${{ env.TARGET_COVERAGE_DIR}}"

run_test.bat is like (depends on where the bat file is, the relative path needs to be updated):
"%1" -batchmode -nographics -logFile "..\..\..\%2\playmode.log" -projectPath "..\..\..\3DMonitoringSystem" -runTests -testPlatform playmode -testResults "..\%2\playmode-results.xml" -debugCodeOptimization -enableCodeCoverage -coverageResultsPath "..\%3" -coverageHistoryPath "..\%2" -coverageOptions "generateAdditionalMetrics;generateHtmlReport;generateHtmlReportHistory;generateBadgeReport;assemblyFilters:-Assembly-CSharp-Editor,-My.Assembly;pathFilters:+*/Assets/_Project/*,+*/Assets/InputSettings/*"