lyndsey-ferguson/fastlane-plugin-test_center

Coverage with multi_scan is not equal to scan coverage.

Opened this issue · 1 comments

New Issue Checklist

  • Updated fastlane-plugin-test_center to the latest version
  • I read the README.md
  • I reviewed the example(s) for the action(s) I am using
  • I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see
  • I have reviewed the Discussions forum to see my question has already been addressed.

Issue Description

Coverage with multi_scan is not equal scan covarage.

We know that there are some differences with fastlane coverage report and xcode coverage report. It sometimes calculates lesser than Xcode. But we expected multi_scan calculates the same percentage with scan.

We have 2500 test cases,

  • Coverage when we run tests on Xcode: %43,6
  • Coverage when we run tests with scan: %43,36
  • Coverage when we run tests with multi_scan: %42,64
lane :run_unit_test_with_scan do
   scan( 
      workspace: "#{PROJECT_SCHEME}.xcworkspace",
      scheme: #{PROJECT_SCHEME},
      disable_concurrent_testing: true,
      prelaunch_simulator: true,
      derived_data_path: "./DerivedData",
      output_types: "html, junit",
      output_directory: "./reports",
      only_testing: ["#{PROJECT_SCHEME_TEST}"],
      code_coverage: true,
      result_bundle: true
   )
end


lane :run_unit_test_with_multi_scan do
   multi_scan( 
      workspace: "#{PROJECT_SCHEME}.xcworkspace",
      scheme: #{PROJECT_SCHEME},
      disable_concurrent_testing: true,
      prelaunch_simulator: false,
      derived_data_path: "./DerivedData",
      output_types: "junit",
      output_directory: "./reports",
      only_testing: ["#{PROJECT_SCHEME_TEST}"],
      code_coverage: true,
      result_bundle: true,
      fail_build: true,
      try_count: 3,
      batch_count: 4,
     parallel_testtrun_count: 4
   )
end

lane :check_code_coverage do |options|
  xcov minimum_coverage_percentage:options[:coverage],
  only_project_targets: true,
  markdown_report:true,
  json_report:true,
  xcov_file_direct_path:".reports/#{PROJECT_SCHEME}.xcresult",
  scheme:PROJECT_SCHEME
end

I want to apologize that I have not responded to this issue. A family health issue is requiring me to narrow my focus on the essentials and I don't have the time I need to focus on the plugin more than a minute. If you can get some other contributors to this project to help with this, I would consider merging a resulting PR.