multi_scan does not find any tests when :only_testing is an array of testTargets
Closed this issue · 17 comments
New Issue Checklist
- Updated
fastlane-plugin-test_centerto the latest version - I read the README.md
- I reviewed the example(s) for the action(s) I am using
- I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see
If you love this fastlane plugin, consider sponsoring it or asking your company to sponsor it. I would really appreciate any
gesture: https://github.com/sponsors/lyndsey-ferguson. 😍
Issue Description
When we upgrade test_center_plugin version to the latest ('3.13.2') tests are not run. It listed all of the tests but it shows following logs:
DEBUG [2020-08-14 11:59:26.81]: > setup_logcollection
DEBUG [2020-08-14 11:59:26.81]: < done in TestCenter::Helper::MultiScanManager.initialize
DEBUG [2020-08-14 11:59:26.81]: Removing pre-existing xcresult bundles:
DEBUG [2020-08-14 11:59:26.81]: /Users/selinunal/Projects/mobile/ipad/fastlane/test_output/report.xcresult
DEBUG [2020-08-14 11:59:26.81]: Fetching available simulator devices
DEBUG [2020-08-14 11:59:27.07]: Results for each test run: []
+-------------------+------+
| multi_scan results |
+-------------------+------+
| result | true |
| total_tests | 0 |
| passing_testcount | 0 |
| failed_testcount | 0 |
| failed_tests | [] |
| total_retry_count | 0 |
| report_files | [] |
+-------------------+------+
Complete output when running fastlane, including the stack trace and command used
[INSERT OUTPUT HERE]
Environment
I will list some gem versions below:
bundler | 2.1.4
xcresult | 0.2.1
xctest_list | 1.2.1
@selin194 thank you for the bug report, sorry that this is happening.
Can you share how you are calling multi_scan? Two things changed since 3.13.0: 1) the method in which testsuites are expanded when given to the :only_testing option, and 2) how :parallel_testrun_count is pinned to the number of tests (i.e. if you only have 2 tests, and ask for 4 parallel test runs, you'll get 2 as 2 parallel test runs are wasted).
I can see that tests are being picked up by these integration checks.
Hey! We are calling multi_scan with only_testingand parallel_testrun_countwith value 2.
Not sure why I can not see our tests are not executed 🤔
Yes, but what values are in :only_testing? Is it an array of testsuites? An array of 'test-targets/testSuites'? An array of 'testSuites/testcases'?
We are calling like only_testing: 'ourTests',
And is that a testSuite?
It is a test target under the app scheme. We have multiple test targets (unit, e2e, snapshot) and we are calling only 1 target at a time. Therefore we are using only_testing option by providing target name. Each target can have multiple test suites.
@selin194 We have a similar design. I have ApplicationUI under our schema and many other different target and when I use only_testing that is not picking up our application tests and if I do not use only_testing, in that case, it is picking up something random unit tests. I tried to use scan and it works fine with scan
Okay, that is something that I hadn't anticipated. It makes sense, for now, can you use 3.13.1 or earlier?
I can try
I think:only_tests doesn't work in 3.13.0
@jambudi you may be experiencing a different problem from @selin194; she found that 3.13.0 did execute her tests successfully.
Can you run your fastlane with the --verbose flag and include the logs in this Issue as well as provide an example of how you are calling multi_scan (please obfuscate any sensitive data).
I will open a different ticket. Thank you.
@selin194 can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?
Pluginfile:
gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-271-only-testing-test-targets-not-expanded"
If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.
Hello @lyndsey-ferguson! I tried with this branch and it seems that it worked and all tests are executed. Will it be fixed? Thanks!