No tests found when passing test suite to multi_scan using the only_testing param
raviTokopedia opened this issue · 3 comments
New Regression Checklist
- Updated
fastlane-plugin-test_centerto the latest version - I read the README.md
- I reviewed the example(s) for the action(s) I am using
- I searched for existing GitHub issues
- I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see
- I have reviewed the Discussions forum to see my question has already been addressed.
If you love this fastlane plugin, consider sponsoring it or asking your company to sponsor it. I would really appreciate any
gesture: https://github.com/sponsors/lyndsey-ferguson.😍
Regression Information
- Breaking version: 3.14.4
- Last working version: 3.14.3
Regression Description
Multiscan started reporting the number of tests 0. I have tried using the latest v3.15 as well but the result is the same.
multi_scan config
multi_scan(
project: 'ios/TokopediaBazel.xcodeproj',
scheme: 'TokopediaUITests',
device: 'iPhone 11',
clean: false,
derived_data_path: './d',
only_testing: tests,
output_types: 'junit',
output_files: 'report.junit',
build_for_testing: true,
result_bundle: true,
fail_build: false,
skip_build: false,
output_directory: 'reports',
configuration: configuration,
prelaunch_simulator: true,
xcargs: xcargs,
disable_concurrent_testing: true,
retry_test_runner_failures: true,
try_count: 3,
testrun_completed_block: test_run_block
)
Complete output when running fastlane, including the stack trace and command used
DEBUG [2021-01-07 14:31:02.45]: Checking if there are any plugins that should be loaded... DEBUG [2021-01-07 14:31:02.45]: Loading 'fastlane-plugin-versioning' plugin INFO [2021-01-07 14:31:02.45]: gem 'fastlane-plugin-versioning' is already installed DEBUG [2021-01-07 14:31:02.46]: Loading 'fastlane-plugin-sentry' plugin INFO [2021-01-07 14:31:02.47]: gem 'fastlane-plugin-sentry' is already installed DEBUG [2021-01-07 14:31:02.47]: Loading 'fastlane-plugin-xchtmlreport' plugin INFO [2021-01-07 14:31:02.47]: gem 'fastlane-plugin-xchtmlreport' is already installed DEBUG [2021-01-07 14:31:02.47]: Loading 'fastlane-plugin-appcenter' plugin INFO [2021-01-07 14:31:02.47]: gem 'fastlane-plugin-appcenter' is already installed DEBUG [2021-01-07 14:31:02.51]: Loading 'fastlane-plugin-test_center' plugin INFO [2021-01-07 14:31:02.51]: gem 'fastlane-plugin-test_center' is already installed +------------------------------+---------+---------------------------------------------------+ | Used plugins | +------------------------------+---------+---------------------------------------------------+ | Plugin | Version | Action | +------------------------------+---------+---------------------------------------------------+ | fastlane-plugin-versioning | 0.4.4 | get_version_number_from_git_branch | | | | increment_build_number_in_xcodeproj | | | | get_version_number_from_plist | | | | get_build_number_from_plist ci_build_number | | | | get_version_number_from_xcodeproj | | | | increment_version_number_in_xcodeproj | | | | increment_version_number_in_plist | | | | get_app_store_version_number get_info_plist_path | | | | get_build_number_from_xcodeproj | | | | increment_build_number_in_plist | | fastlane-plugin-sentry | 1.6.0 | sentry_upload_dsym sentry_set_commits | | | | sentry_upload_sourcemap sentry_finalize_release | | | | sentry_upload_file sentry_upload_proguard | | | | sentry_create_release | | fastlane-plugin-xchtmlreport | 0.1.1 | xchtmlreport | | fastlane-plugin-appcenter | 1.9.0 | appcenter_fetch_version_number | | | | appcenter_fetch_devices appcenter_upload | | fastlane-plugin-test_center | 3.15.0 | suppressed_tests suppress_tests | | | | tests_from_xcresult suppress_tests_from_junit | | | | tests_from_xctestrun quit_core_simulator_service | | | | collate_test_result_bundles tests_from_junit | | | | multi_scan test_options_from_testplan | | | | collate_html_reports collate_junit_reports | | | | collate_json_reports testplans_from_scheme | | | | collate_xcresults | +------------------------------+---------+---------------------------------------------------+
DEBUG [2021-01-07 14:31:03.34]: Using tests: TokopediaPerformanceTests
INFO [2021-01-07 14:31:03.35]: ----------------------------------------
INFO [2021-01-07 14:31:03.35]: --- Step: Verifying fastlane version ---
INFO [2021-01-07 14:31:03.35]: ----------------------------------------
INFO [2021-01-07 14:31:03.35]: Your fastlane version 2.160.0 matches the minimum requirement of 1.90.0 ✅
INFO [2021-01-07 14:31:03.35]: ------------------------------
INFO [2021-01-07 14:31:03.35]: --- Step: default_platform ---
INFO [2021-01-07 14:31:03.36]: ------------------------------
INFO [2021-01-07 14:31:03.36]: Driving the lane 'ios performance_test' 🚀
INFO [2021-01-07 14:31:03.36]: -------------------
INFO [2021-01-07 14:31:03.36]: --- Step: is_ci ---
INFO [2021-01-07 14:31:03.36]: -------------------
INFO [2021-01-07 14:31:03.36]: ---------------------------
INFO [2021-01-07 14:31:03.36]: --- Step: setup_jenkins ---
INFO [2021-01-07 14:31:03.36]: ---------------------------
WARN [2021-01-07 14:31:03.36]: Not executed by Continuous Integration system.
INFO [2021-01-07 14:31:03.36]: -----------------------------
INFO [2021-01-07 14:31:03.36]: --- Step: last_git_commit ---
INFO [2021-01-07 14:31:03.36]: -----------------------------
INFO [2021-01-07 14:31:03.44]: ------------------------
INFO [2021-01-07 14:31:03.44]: --- Step: git_branch ---
INFO [2021-01-07 14:31:03.44]: ------------------------
INFO [2021-01-07 14:31:03.45]: Removing apps in device with udid
INFO [2021-01-07 14:31:03.46]: ------------------------
INFO [2021-01-07 14:31:03.46]: --- Step: multi_scan ---
INFO [2021-01-07 14:31:03.46]: ------------------------
WARN [2021-01-07 14:31:03.46]: As of Xcode 11, test_result bundles created in the output directory are actually symbolic links to an xcresult bundle+----------------------------+--------------------------------------------------+
| Summary for multi_scan (test_center v3.15.0) |
+----------------------------+--------------------------------------------------+
| retry_test_runner_failures | true |
| try_count | 3 |
| testrun_completed_block | #<Proc:0x00007fabc9c05ec8@Fastfile:223 (lambda)> |
+----------------------------+--------------------------------------------------+ERROR [2021-01-07 14:31:03.48]: Using deprecated option: '--custom_report_file_name' (Use
--output_filesinstead)
DEBUG [2021-01-07 14:31:03.48]: Building the project in preparation for multi_scan testing
ERROR [2021-01-07 14:31:03.48]: Using deprecated option: '--custom_report_file_name' (Use--output_filesinstead)
WARN [2021-01-07 14:31:03.54]: Resolving Swift Package Manager dependencies...
INFO [2021-01-07 14:31:03.54]: $ xcodebuild -resolvePackageDependencies -scheme TokopediaUITests -project ios/TokopediaBazel.xcodeproj -configuration Debug -derivedDataPath ./d
INFO [2021-01-07 14:31:04.15]: ▸ Command line invocation:
INFO [2021-01-07 14:31:04.15]: ▸ /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -resolvePackageDependencies -scheme TokopediaUITests -project ios/TokopediaBazel.xcodeproj -configuration Debug -derivedDataPath ./d
INFO [2021-01-07 14:31:04.15]: ▸ User defaults from command line:
INFO [2021-01-07 14:31:04.15]: ▸ IDEDerivedDataPathOverride = /Users/tokopedia/Projects/ios-tokopedia/d
INFO [2021-01-07 14:31:20.10]: ▸ resolved source packages:
INFO [2021-01-07 14:31:20.14]: $ xcodebuild -showBuildSettings -scheme TokopediaUITests -project ios/TokopediaBazel.xcodeproj -configuration Debug -derivedDataPath ./d
DEBUG [2021-01-07 14:31:48.62]: Fetching available simulator devices+------------------------------------------------+-----------------------------------------------------------------+
| Summary for scan 2.160.0 |
+------------------------------------------------+-----------------------------------------------------------------+
| project | ios/TokopediaBazel.xcodeproj |
| scheme | TokopediaUITests |
| device | iPhone 11 |
| clean | false |
| derived_data_path | ./d |
| only_testing | ["TokopediaUITests/TokopediaPerformanceTests"] |
| output_types | junit |
| output_files | report.junit |
| build_for_testing | true |
| fail_build | false |
| output_directory | reports |
| configuration | Debug |
| prelaunch_simulator | true |
| xcargs | -UseNewBuildSystem=YES OTHER_SWIFT_FLAGS='-DSIGNPOST_TRACKING' |
| | GCC_PREPROCESSOR_DEFINITIONS='$GCC_PREPROCESSOR_DEFINITIONS |
| | SIGNPOST_TRACKING=1' -parallel-testing-enabled NO |
| disable_concurrent_testing | true |
| skip_detect_devices | false |
| force_quit_simulator | false |
| reset_simulator | false |
| disable_slide_to_type | true |
| reinstall_app | false |
| app_identifier | |
| open_report | false |
| buildlog_path | ~/Library/Logs/scan |
| should_zip_build_products | false |
| use_clang_report_name | false |
| slack_use_webhook_configured_username_and_icon | false |
| slack_username | fastlane |
| slack_icon_url | https://fastlane.tools/assets/img/fastlane_icon.png |
| skip_slack | false |
| slack_only_on_failure | false |
| xcodebuild_command | env NSUnbufferedIO=YES xcodebuild |
| include_simulator_logs | false |
| skip_build | false |
| xcode_path | /Applications/Xcode.app |
+------------------------------------------------+-----------------------------------------------------------------+DEBUG [2021-01-07 14:31:48.87]: Before building, removing pre-existing xctestrun files: ["./d/Build/Products/TokopediaUITests_iphonesimulator14.3-x86_64.xctestrun"]
DEBUG [2021-01-07 14:31:48.87]: Fetching available simulator devices
INFO [2021-01-07 14:31:49.10]: Disabling 'Slide to Type' iPhone 11
INFO [2021-01-07 14:31:49.10]: $ /usr/libexec/PlistBuddy -c "Add :KeyboardContinuousPathEnabled bool false" /Users/tokopedia/Library/Developer/CoreSimulator/Devices/1D9D3959-FB52-4D00-9722-AD894101F73C/data/Library/Preferences/com.apple.keyboard.ContinuousPath.plist >/dev/null 2>&1
INFO [2021-01-07 14:31:49.10]: Booting iPhone 11
INFO [2021-01-07 14:31:49.64]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme TokopediaUITests -project ios/TokopediaBazel.xcodeproj -configuration Debug -derivedDataPath ./d -destination 'platform=iOS Simulator,id=1D9D3959-FB52-4D00-9722-AD894101F73C' -disable-concurrent-testing -UseNewBuildSystem=YES OTHER_SWIFT_FLAGS='-DSIGNPOST_TRACKING' GCC_PREPROCESSOR_DEFINITIONS='$GCC_PREPROCESSOR_DEFINITIONS SIGNPOST_TRACKING=1' -parallel-testing-enabled NO -only-testing:TokopediaUITests/TokopediaPerformanceTests build-for-testing | tee '/Users/tokopedia/Library/Logs/scan/TokopediaUITests-TokopediaUITests.log' | xcpretty --report junit --output '/Users/tokopedia/Projects/ios-tokopedia/reports/report.junit' --report junit --output '/var/folders/n2/_rzdhc7d7_j7yp7vblndg3xr0000gn/T/junit_report20210107-50829-nsyz1b'
INFO [2021-01-07 14:31:49.64]: ▸ Loading...
INFO [2021-01-07 14:33:19.98]: ▸ Skipping code signing because the target does not have an Info.plist file and one is not being generated automatically. (in target 'Pods_nanopb-nanopb' from project 'TokopediaBazel')
INFO [2021-01-07 14:33:19.98]: ▸ Skipping code signing because the target does not have an Info.plist file and one is not being generated automatically. (in target 'Inbox_InboxSharedModels' from project 'TokopediaBazel')
INFO [2021-01-07 14:33:19.98]: ▸ Skipping code signing because the target does not have an Info.plist file and one is not being generated automatically. (in target 'Pods_NWPusher-NWPusher' from project 'TokopediaBazel')
INFO [2021-01-07 14:33:20.50]: ▸ Test build Succeeded
INFO [2021-01-07 14:33:22.02]: Successfully sent Slack notification
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+DEBUG [2021-01-07 14:33:22.03]: After building, found xctestrun files ["./d/Build/Products/TokopediaUITests_iphonesimulator14.3-x86_64.xctestrun"] (choosing 1st)
DEBUG [2021-01-07 14:33:22.03]: Removing report files generated by the build
DEBUG [2021-01-07 14:33:22.03]: reports/report.junit+-------------------+------+
| multi_scan results |
+-------------------+------+
| result | true |
| total_tests | 0 |
| passing_testcount | 0 |
| failed_testcount | 0 |
| total_retry_count | 0 |
+-------------------+------+INFO [2021-01-07 14:33:22.03]: Successfully generated documentation at path '/Users/tokopedia/Projects/ios-tokopedia/fastlane/README.md'
+------+----------------------------+-------------+
| fastlane summary |
+------+----------------------------+-------------+
| Step | Action | Time (in s) |
+------+----------------------------+-------------+
| 1 | Verifying fastlane version | 0 |
| 2 | default_platform | 0 |
| 3 | is_ci | 0 |
| 4 | setup_jenkins | 0 |
| 5 | last_git_commit | 0 |
| 6 | git_branch | 0 |
| 7 | multi_scan | 138 |
+------+----------------------------+-------------++-------------+--------------+----------------+
| Plugin updates available |
+-------------+--------------+----------------+
| Plugin | Your Version | Latest Version |
+-------------+--------------+----------------+
| sentry | 1.6.0 | 1.7.0 |
| appcenter | 1.9.0 | 1.11.0 |
| test_center | 3.15.0 | 3.15.1 |
+-------------+--------------+----------------+
INFO [2021-01-07 14:33:22.05]: To update all plugins, just run
INFO [2021-01-07 14:33:22.05]: $ bundle exec fastlane update_pluginsINFO [2021-01-07 14:33:22.05]: fastlane.tools finished successfully 🎉
INFO [2021-01-07 14:33:29.32]: Shutting down 1D9D3959-FB52-4D00-9722-AD894101F73C
Environment
🚫 fastlane environment 🚫
Stack
Key Value OS 11.1 Ruby 2.6.3 Bundler? true Git git version 2.24.3 (Apple Git-128) Installation Source ~/Projects/ios-tokopedia/vendor/bundle/ruby/2.6.0/bin/fastlane Host macOS 11.1 (20C69) Ruby Lib Dir /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib OpenSSL Version LibreSSL 2.8.3 Is contained false Is homebrew false Is installed via Fabric.app false Xcode Path /Applications/Xcode.app/Contents/Developer/ Xcode Version 12.3 System Locale
Error No Locale with UTF8 found 🚫 fastlane gems
Gem Version Update-Status fastlane 2.160.0 🚫 Update available Loaded fastlane plugins:
Plugin Version Update-Status fastlane-plugin-versioning 0.4.4 ✅ Up-To-Date fastlane-plugin-sentry 1.7.0 ✅ Up-To-Date fastlane-plugin-xchtmlreport 0.1.1 ✅ Up-To-Date fastlane-plugin-appcenter 1.11.0 ✅ Up-To-Date fastlane-plugin-test_center 3.15.1 ✅ Up-To-Date Loaded gems
Gem Version did_you_mean 1.3.0 bundler 2.1.4 rake 13.0.3 CFPropertyList 3.0.3 concurrent-ruby 1.1.7 i18n 0.9.5 minitest 5.14.2 thread_safe 0.3.6 tzinfo 1.2.7 activesupport 4.2.11.3 public_suffix 4.0.6 addressable 2.7.0 atomos 0.1.3 aws-eventstream 1.1.0 aws-partitions 1.414.0 aws-sigv4 1.2.2 jmespath 1.4.0 aws-sdk-core 3.110.0 aws-sdk-kms 1.40.0 aws-sdk-s3 1.87.0 babosa 1.0.4 claide 1.0.3 colored2 3.1.2 cork 0.3.0 nap 1.1.0 open4 1.3.4 claide-plugins 0.9.2 fuzzy_match 2.0.4 cocoapods-core 1.5.3 cocoapods-deintegrate 1.0.4 cocoapods-downloader 1.4.0 cocoapods-plugins 1.0.0 cocoapods-search 1.0.0 cocoapods-stats 1.1.0 netrc 0.11.0 cocoapods-trunk 1.5.0 cocoapods-try 1.2.0 escape 0.0.4 fourflusher 2.0.1 gh_inspector 1.1.3 molinillo 0.6.6 ruby-macho 1.4.0 nanaimo 0.3.0 xcodeproj 1.18.0 cocoapods 1.5.3 cocoapods-static-swift-framework 0.5 cocoapods-transitive-dependencies 0.0.1 colored 1.2 colorize 0.8.1 highline 1.7.10 commander-fastlane 4.4.6 faraday-net_http 1.0.0 multipart-post 2.0.0 ruby2_keywords 0.0.2 faraday 1.3.0 faraday-http-cache 2.2.0 rchardet 1.8.0 git 1.7.0 rexml 3.2.4 kramdown 2.3.0 kramdown-parser-gfm 1.1.0 no_proxy_fix 0.1.2 sawyer 0.8.2 octokit 4.18.0 unicode-display_width 1.7.0 terminal-table 1.8.0 danger 8.0.5 danger-plugin-api 1.0.0 danger-swiftformat 0.7.0 thor 0.20.3 danger-swiftlint 0.23.0 declarative 0.0.20 declarative-option 0.1.0 digest-crc 0.6.3 unf_ext 0.0.7.7 unf 0.1.4 domain_name 0.5.20190701 dotenv 2.7.6 emoji_regex 3.2.1 excon 0.78.1 http-cookie 1.0.3 faraday-cookie_jar 0.0.7 faraday_middleware 1.0.0 fastimage 2.2.1 jwt 2.2.2 memoist 0.16.2 multi_json 1.15.0 os 1.1.1 signet 0.14.0 googleauth 0.14.0 httpclient 2.8.3 mini_mime 1.0.2 uber 0.1.0 representable 3.0.4 retriable 3.1.2 google-api-client 0.38.0 google-cloud-env 1.4.0 google-cloud-errors 1.0.1 google-cloud-core 1.5.0 google-cloud-storage 1.29.2 json 2.5.1 mini_magick 4.11.0 plist 3.6.0 rubyzip 2.3.0 security 0.1.3 naturally 2.2.0 simctl 1.6.8 slack-notifier 2.3.2 terminal-notifier 2.0.0 tty-screen 0.8.1 tty-cursor 0.7.1 tty-spinner 0.9.3 word_wrap 1.0.0 rouge 2.0.7 xcpretty 0.3.0 xcpretty-travis-formatter 1.0.1 fastlane-plugin-appcenter 1.11.0 fastlane-plugin-sentry 1.7.0 trainer 0.9.1 xctest_list 1.2.1 fastlane-plugin-test_center 3.15.1 fastlane-plugin-versioning 0.4.4 fastlane-plugin-xchtmlreport 0.1.1 ffi 1.13.1 mustache 1.1.1 redcarpet 3.5.0 sassc 2.4.0 sqlite3 1.4.2 liferaft 0.0.6 xcinvoke 0.3.0 jazzy 0.12.0 generated on: 2021-01-07
Kind of a duplicate of #293, but it doesn't work for me.
Can you remove this option:
build_for_testing: true
From your call to multi_scan and try again. Before, when you passed this, it would also test, which is the opposite of what you're asking. When you pass in build_for_testing: true, that's all it should do.
Please let me know the results.
Closing this due to lack of response and data