No tests found when using multi_scan with XCUITest
craig-jamieson opened this issue · 22 comments
New Issue Checklist
- Updated
fastlane-plugin-test_centerto the latest version - I read the README.md
- I reviewed the example(s) for the action(s) I am using
- I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see
- I have reviewed the Discussions forum to see my question has already been addressed.
Issue Description
I've tried some of the suggestions here, but I'm still encountering the same issue. I'm trying to use multi_scan with a project using xcuitest:
multi_scan(
workspace: "project.xcworkspace",
scheme: "XXXX",
try_count: 3,
fail_build: false,
parallel_testrun_count: 4,
clean: clean,
configuration: "S101",
destination: "platform=iOS Simulator,OS=#{IOS_VERSION},name=#{IPHONE_VERSION}",
reset_simulator: true,
include_simulator_logs: false,
skip_slack: true,
code_coverage: true,
only_testing: ["UITests"],
xcargs: "-jobs `sysctl hw.ncpu | awk '{print 2*$2}'`",
derived_data_path: DERIVED_DATA_FOLDER,
output_directory: TEST_RESULTS_FOLDER,
output_files: "report.html,junit.xml"
)
rescue => ex
report_result = build_failing_tests_report(report_identifier: "Report ID", test_results_folder: TEST_RESULTS_FOLDER)
total_tests_count = 0
total_failed_tests = 0
for failing_test_suite in report_result
if failing_test_suite[:failures] == "0"
next
end
And the result is the next ("Test build Succeeded" but zero tests)
[14:58:32]: ▸ Test build Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+
[14:58:35]: No tests found in 'test-output/derived-data/Build/Products/Stage Debug-iphonesimulator/xxx.app/PlugIns/xxx.xctest'!
[14:58:35]: Is the Build Setting, `ENABLE_TESTABILITY` enabled for the test target XXXX?
[14:58:35]: If your Swift test method names use a prefix other than `test`, consider setting `:swift_test_prefix`.
[14:58:37]: No tests found in 'test-output/derived-data/Build/Products/Stage Debug-iphonesimulator/XXX.app/PlugIns/XXX.xctest'!
[14:58:37]: Is the Build Setting, `ENABLE_TESTABILITY` enabled for the test target XXXX?
[14:58:37]: If your Swift test method names use a prefix other than `test`, consider setting `:swift_test_prefix`.
[14:58:38]: No tests found in 'test-output/derived-data/Build/Products/Stage Debug-iphonesimulator/XXX.app/PlugIns/XXX.xctest'!
[14:58:38]: Is the Build Setting, `ENABLE_TESTABILITY` enabled for the test target XXXXX?
[14:58:38]: If your Swift test method names use a prefix other than `test`, consider setting `:swift_test_prefix`.
[14:58:38]: No tests found in 'test-output/derived-data/Build/Products/Stage Debug-iphonesimulator/XXXX.app/PlugIns/XXX.xctest'!
[14:58:38]: Is the Build Setting, `ENABLE_TESTABILITY` enabled for the test target XXX?
[14:58:38]: If your Swift test method names use a prefix other than `test`, consider setting `:swift_test_prefix`.
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
[14:58:40]: ----------------------------------------
[14:58:40]: --- Step: build_failing_tests_report ---
[14:58:40]: ----------------------------------------
/Users/distiller/i/test-output/results/junit.xml
+---------------------------+---------------------------------------------------------------+
| Lane Context |
+---------------------------+---------------------------------------------------------------+
| DEFAULT_PLATFORM | ios |
| PLATFORM_NAME | ios |
| LANE_NAME | ios e2e_tests |
| KEYCHAIN_PATH | ~/Library/Keychains/path |
| ORIGINAL_DEFAULT_KEYCHAIN | "/Users/distiller/Library/Keychains/path" |
+---------------------------+---------------------------------------------------------------+
[14:58:40]: No output file found
+------+----------------------------+-------------+
| fastlane summary |
+------+----------------------------+-------------+
| Step | Action | Time (in s) |
+------+----------------------------+-------------+
| 1 | Verifying fastlane version | 0 |
| 2 | default_platform | 0 |
| 3 | setup_circle_ci | 0 |
| 4 | ensure_xcode_version | 2 |
| 💥 | multi_scan | 764 |
| 💥 | build_failing_tests_report | 0 |
+------+----------------------------+-------------+
+------------+--------------+----------------+
| Plugin updates available |
+------------+--------------+----------------+
| Plugin | Your Version | Latest Version |
+------------+--------------+----------------+
| versioning | 0.4.5 | 0.5.0 |
+------------+--------------+----------------+
[14:58:40]: To update all plugins, just run
[14:58:40]: $ bundle exec fastlane update_plugins
[14:58:40]: fastlane finished with errors
[!] No output file found
ENABLE_TESTABILITY is enabled.
Summary for scan
+------------------------------------------------+-----------------------------------------------------+
| Summary for scan 2.187.0 |
+------------------------------------------------+-----------------------------------------------------+
| workspace | XXX.xcworkspace |
| scheme | XXX Stage |
| fail_build | false |
| clean | false |
| configuration | Stage Debug |
| reset_simulator | true |
| skip_slack | true |
| code_coverage | true |
| only_testing | ["UITests"] |
| xcargs | -jobs `sysctl hw.ncpu | awk '{print 2*$2}'` |
| derived_data_path | test-output/derived-data |
| output_directory | test-output/results |
| output_files | report.html,junit.xml |
| output_types | html,junit |
| skip_detect_devices | false |
| ensure_devices_found | false |
| force_quit_simulator | false |
| disable_slide_to_type | true |
| reinstall_app | false |
| app_identifier | Prod |
| open_report | false |
| buildlog_path | artifacts/buildlogs/scan |
| should_zip_build_products | false |
| output_xctestrun | false |
| use_clang_report_name | false |
| disable_concurrent_testing | false |
| build_for_testing | true |
| slack_use_webhook_configured_username_and_icon | false |
| slack_username | fastlane |
| slack_icon_url | https://fastlane.tools/assets/img/fastlane_icon.png |
| slack_only_on_failure | false |
| xcodebuild_command | env NSUnbufferedIO=YES xcodebuild |
| skip_package_dependencies_resolution | false |
| disable_package_automatic_updates | false |
| use_system_scm | false |
| number_of_retries | 0 |
| include_simulator_logs | false |
| skip_build | false |
| xcode_path | /Applications/Xcode-12.5.1.app
Environment
🚫 fastlane environment 🚫
Stack
Key Value OS 11.4 Ruby 2.6.4 Bundler? true Git git version 2.26.2 Installation Source ~/ios-app/vendor/cache/ruby/2.6.0/bin/fastlane Host macOS 11.4 (20F71) Ruby Lib Dir ~/.rbenv/versions/2.6.4/lib OpenSSL Version OpenSSL 1.1.1d 10 Sep 2019 Is contained false Is homebrew false Is installed via Fabric.app false Xcode Path /Applications/Xcode.app/Contents/Developer/ Xcode Version 12.5.1 Swift Version 5.4.2 System Locale
Variable Value LANG en_GB.UTF-8 ✅ LC_ALL LANGUAGE fastlane files:
`./fastlane/Fastfile`
# This file contains the fastlane.tools configuration # You can find the documentation at https://docs.fastlane.tools # # For a list of all available actions, check out # # https://docs.fastlane.tools/actions # require 'net/http' require 'uri' fastlane_require 'yaml' fastlane_version "2.188.0" default_platform(:ios) platform :ios do before_all do # Allow up to 6 attempts for starting xcodebuild ENV["FASTLANE_XCODEBUILD_SETTINGS_RETRIES"] = "5" # Prepare CircleCi configuration, including a usable Keychain setup_circle_ci # This uses the .xcode-version file in the project directory to ensure we # use a consistent Xcode version to ship and on the CI to test. # If upgrading to a new Xcode, change the `.xcode-version` file. ensure_xcode_version end import("./Fastfiles/Setup.rb") import("./Fastfiles/Testing.rb") import("./Fastfiles/Release.rb") import("./Fastfiles/EmergencyRelease.rb") import("./Fastfiles/Utilities.rb") import("./Fastfiles/Build.rb") # CircleCI exclusive lanes. Only imported if IS_CI_MACHINE if ENV["IS_CI_MACHINE"] == "TRUE" import("./Fastfiles/CircleCI.rb") end end # Path resolvers def build_output_path File.expand_path("../#{BUILD_OUTPUT_FOLDER}") end def ipa_file_path File.join(build_output_path, IPA_FILENAME) end def dsyms_file_path File.join(build_output_path, DSYM_FILENAME) end`./fastlane/Appfile`
app_identifier "Prod" # The bundle identifier of your app apple_id "" # Your Apple email address # For more information about the Appfile, see: # https://docs.fastlane.tools/advanced/#appfilefastlane gems
Gem Version Update-Status fastlane 2.188.0 🚫 Update available Loaded fastlane plugins:
Plugin Version Update-Status fastlane-plugin-aws_s3 2.0.2 🚫 Update available fastlane-plugin-firebase_app_distribution 0.2.9 ✅ Up-To-Date fastlane-plugin-versioning 0.4.5 🚫 Update available fastlane-plugin-test_center 3.15.3 ✅ Up-To-Date Loaded gems
Gem Version did_you_mean 1.3.0 bundler 2.1.4 rake 13.0.6 CFPropertyList 3.0.3 concurrent-ruby 1.1.9 i18n 1.8.10 minitest 5.14.4 thread_safe 0.3.6 tzinfo 1.2.9 activesupport 5.2.6 public_suffix 4.0.6 addressable 2.8.0 httpclient 2.8.3 json 2.5.1 algoliasearch 1.27.5 rubyzip 2.3.2 apktools 0.7.4 artifactory 3.0.15 atomos 0.1.3 aws-eventstream 1.1.1 aws-partitions 1.479.0 aws-sigv4 1.2.4 jmespath 1.4.0 aws-sdk-core 3.117.0 aws-sdk-kms 1.44.0 aws-sdk-s3 1.96.2 babosa 1.0.4 claide 1.0.3 colored2 3.1.2 cork 0.3.0 nap 1.1.0 open4 1.3.4 claide-plugins 0.9.2 fuzzy_match 2.0.4 netrc 0.11.0 ffi 1.15.3 ethon 0.14.0 typhoeus 1.4.0 cocoapods-core 1.10.1 cocoapods-deintegrate 1.0.4 cocoapods-downloader 1.4.0 cocoapods-plugins 1.0.0 cocoapods-search 1.0.0 cocoapods-trunk 1.5.0 cocoapods-try 1.2.0 escape 0.0.4 fourflusher 2.3.1 gh_inspector 1.1.3 molinillo 0.6.6 ruby-macho 1.4.0 nanaimo 0.3.0 rexml 3.2.5 xcodeproj 1.20.0 cocoapods 1.10.1 colored 1.2 colorize 0.8.1 highline 2.0.3 commander 4.6.0 faraday-em_http 1.0.0 faraday-em_synchrony 1.0.0 faraday-excon 1.1.0 faraday-httpclient 1.0.1 faraday-net_http 1.0.1 faraday-net_http_persistent 1.2.0 faraday-patron 1.0.0 multipart-post 2.0.0 ruby2_keywords 0.0.5 faraday 1.5.1 faraday-http-cache 2.2.0 rchardet 1.8.0 git 1.8.1 kramdown 2.3.1 kramdown-parser-gfm 1.1.0 no_proxy_fix 0.1.2 sawyer 0.8.2 octokit 4.21.0 unicode-display_width 1.7.0 terminal-table 1.8.0 danger 8.2.3 thor 0.20.3 danger-swiftlint 0.24.5 declarative 0.0.20 digest-crc 0.6.4 unf_ext 0.0.7.7 unf 0.1.4 domain_name 0.5.20190701 dotenv 2.7.6 emoji_regex 3.2.2 excon 0.85.0 http-cookie 1.0.4 faraday-cookie_jar 0.0.7 faraday_middleware 1.0.0 fastimage 2.2.4 jwt 2.2.3 memoist 0.16.2 multi_json 1.15.0 os 1.1.1 signet 0.15.0 googleauth 0.16.2 mini_mime 1.1.0 trailblazer-option 0.1.1 uber 0.1.0 representable 3.1.1 retriable 3.1.2 webrick 1.7.0 google-apis-core 0.4.1 google-apis-androidpublisher_v3 0.9.0 google-apis-playcustomapp_v1 0.5.0 google-apis-iamcredentials_v1 0.6.0 google-apis-storage_v1 0.6.0 google-cloud-env 1.5.0 google-cloud-errors 1.1.0 google-cloud-core 1.6.0 google-cloud-storage 1.34.1 mini_magick 4.11.0 naturally 2.2.1 plist 3.6.0 security 0.1.3 simctl 1.6.8 terminal-notifier 2.0.0 tty-screen 0.8.1 tty-cursor 0.7.1 tty-spinner 0.9.3 word_wrap 1.0.0 rouge 2.0.7 xcpretty 0.3.0 xcpretty-travis-formatter 1.0.1 mime-types-data 3.2021.0704 mime-types 3.3.1 fastlane-plugin-aws_s3 2.0.2 fastlane-plugin-firebase_app_distribution 0.2.9 trainer 0.9.1 xctest_list 1.2.1 fastlane-plugin-test_center 3.15.3 fastlane-plugin-versioning 0.4.5 mini_portile2 2.5.3 racc 1.5.2 nokogiri 1.11.7 xcode-build-times 0.1.1 xcode-install 2.6.2 generated on: 2021-08-20
Hello @craig-jamieson you didn't explicitly mention it, so I have to ask, did you follow the suggestion to:
[14:58:35]: No tests found in 'test-output/derived-data/Build/Products/Stage Debug-iphonesimulator/xxx.app/PlugIns/xxx.xctest'!
[14:58:35]: Is the Build Setting, `ENABLE_TESTABILITY` enabled for the test target XXXX?
?
Hello @craig-jamieson you didn't explicitly mention it, so I have to ask, did you follow the suggestion to:
[14:58:35]: No tests found in 'test-output/derived-data/Build/Products/Stage Debug-iphonesimulator/xxx.app/PlugIns/xxx.xctest'! [14:58:35]: Is the Build Setting, `ENABLE_TESTABILITY` enabled for the test target XXXX??
Hello @lyndsey-ferguson, thanks for the reply
Yes, I've enabled testability for the appropriate test target
Okay, the next thing I have to ask is for you to run your lane, but this time with the --verbose flag and include the logs.
Okay, the next thing I have to ask is for you to run your lane, but this time with the
--verboseflag and include the logs.
Thanks, see attached
e2e1.odt
The logs indicate that the plugin is crashing in the runner and that exception is caught in your Fastfile. The logs do not provide enough information to indicate where in the runner it is crashing. I've made some modifications to provide more logs.
Can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?
Pluginfile:
gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-353-exception-when-starting-tests"
Thank you!
See attached
e2et.txt
For my reference, this is the guts of where this is failing:
DEBUG [2021-08-31 10:09:10.00]: < setup_testcollector
DEBUG [2021-08-31 10:09:10.00]: > setup_logcollection
DEBUG [2021-08-31 10:09:10.00]: < done in TestCenter::Helper::MultiScanManager.initialize
DEBUG [2021-08-31 10:09:10.00]: > remove_preexisting_test_result_bundles
DEBUG [2021-08-31 10:09:10.00]: > remote_preexisting_xcresult_bundles
DEBUG [2021-08-31 10:09:10.00]: > setup_testcollector
FAIL HERE :(
@craig-jamieson, I just noticed that you have this -jobs sysctl hw.ncpu | awk '{print 2*$2}' for the xcargs parameter. I imagine that you're trying to achieve parallel jobs with this, but it may interfere with the logic that I've built into multi_scan's parallel_testrun_count....just food for thought as I add more logs.
Can you modify your Pluginfile per my instructions below, remove your Gemfile.lock file, run bundle install, and then run your fastlane again (with the --verbose flag)?
Pluginfile:
gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-353-exception-when-starting-tests"
Please attach the logs. Thanks!
See logs attached.
For my own reference, it is failing somewhere here:
DEBUG [2021-09-07 10:06:28.38]: < setup_testcollector
DEBUG [2021-09-07 10:06:28.38]: > setup_logcollection
DEBUG [2021-09-07 10:06:28.38]: < done in TestCenter::Helper::MultiScanManager.initialize
DEBUG [2021-09-07 10:06:28.41]: > remove_preexisting_test_result_bundles
DEBUG [2021-09-07 10:06:28.41]: > remove_preexisting_xcresult_bundles
DEBUG [2021-09-07 10:06:28.41]: About to run_test_batches
DEBUG [2021-09-07 10:06:28.41]: > setup_testcollector
DEBUG [2021-09-07 10:06:28.41]: checking if the platform is :ios_simulator
DEBUG [2021-09-07 10:06:28.41]: working with scan_destinations: ["platform=iOS Simulator,OS=14.5,name=iPhone 11 Pro"]
INFO [2021-09-07 10:06:28.76]: ----------------------------------------
INFO [2021-09-07 10:06:28.76]: --- Step: build_failing_tests_report ---
INFO [2021-09-07 10:06:28.76]: ----------------------------------------
Okay, so I believe I know what the error is: the :destination is being provided and most of the users do not pass it in directly (or if they do, they provide the udid value). The code at this point is looking for a udid value to identify the unique simulator. The code should detect when it is missing and look it up. I also see that the old way of specifying the simulator with the :device parameter, i.e. "iPhone 11 Pro (14.5)" is now not working with the underlying scan command.
@craig-jamieson for now, do you have a way of getting the udid for the iPhone 11 Pro you want to use and provide that as an additional text in the :destination parameter? ("platform=iOS SImulator,OS=14.5,name=iPhone 11 Pro,udid=")
Thanks @lyndsey-ferguson for the info
I'm having some trouble using the udid as a parameter at the moment, is there anything else you can suggest as a workaround?
You should be able to get the udid value via:
xcrun simctl list devices --json
https://www.iosdev.recipes/simctl/
Find the iPhone 11 Pro and copy the udid of the device you want.
If that is already what you are doing, can you describe what the trouble is that you're having? I don't know of another work around at the moment, I will let my subconscious chew on it though.
@craig-jamieson bump
@lyndsey-ferguson I made some progress, I was able to add the UDID as a parameter but the tests are still failing to execute, see below excerpt from logs:
[worker 5] INFO [2021-09-13 18:05:54.12]: ▸ ** TEST EXECUTE FAILED **
[worker 5] 2021-09-13 18:05:46.985 xcodebuild[99117:9444605] [MT] IDETestOperationsObserverDebug: 246.121 elapsed -- Testing started completed.
[worker 5] 2021-09-13 18:05:46.985 xcodebuild[99117:9444605] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[worker 5] 2021-09-13 18:05:46.985 xcodebuild[99117:9444605] [MT] IDETestOperationsObserverDebug: 246.121 sec, +246.121 sec -- end
[worker 5] Testing failed:
[worker 5] XXXUITests:
[worker 5] The test runner encountered an error (Failed to prepare device 'iPhone 12 Clone 5 TestCenter::Helper::MultiScanManager::SimulatorHelper<70349616173520>' for impending launch. If you believe this error represents a bug, please attach the result bundle at /var/folders/12/bp7yqvlj1v70pk6nm_4w17900000gn/T/derived_data_path20210913-96407-1jrvf7q-worker-5/Logs/Test/Test-Transient Testing-2021.09.13_18-01-40-+0100.xcresult. (Underlying Error: Unable to boot the Simulator. launchd failed to respond. (Underlying Error: Failed to start launchd_sim: could not bind to session, launchd_sim may have crashed or quit responding)))
[worker 5]
[worker 5] ** TEST EXECUTE FAILED **
[worker 5] ERROR [2021-09-13 18:05:54.46]: Exit status: 65
[worker 5] DEBUG [2021-09-13 18:05:54.58]: Batch #5 incrementing retry count to 1
[worker 5] DEBUG [2021-09-13 18:05:54.59]: Scan unable to test for batch #5
[worker 5] ERROR [2021-09-13 18:05:54.59]:
[worker 5] Error building the application. See the log above.
Hm. Can you include more logs. Also, show me the line for the device that you found after running:
xcrun simctl list devices --json
🙏
I see that the xargs flag that you're passing (-jobs sysctl hw.ncpu | awk '{print 2*$2}'), could be splitting the command line to xcodebuild so that it is not passing the tests to run, can you remove that option and retry the command?
Sorry for the delay, the tests are now being executed 🙌
I'm experiencing some other issues with circleci but I'll raise that separately after I've investigated
Okay, great! I'll put this issue back on the to-do pile now that you're unblocked.
I want to apologize that I have not responded to this issue. A family health issue is requiring me to narrow my focus on the essentials and I don't have the time I need to focus on the plugin more than a minute. If you can get some other contributors to this project to help with this, I would consider merging a resulting PR.