multi_scan The flag -testPlan <name> cannot be used error when test_without_building = true

Question

multi_scan The flag -testPlan <name> cannot be used error when test_without_building = true

kitskomina opened this issue 4 years ago · 38 comments

New Issue Checklist

Updated fastlane-plugin-test_center to the latest version
I read the README.md
I reviewed the example(s) for the action(s) I am using
I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see

If you love this fastlane plugin, consider sponsoring it or asking your company to sponsor it. I would really appreciate any
gesture: https://github.com/sponsors/lyndsey-ferguson. 😍

Issue Description

When I try to run tests for test plan form prebuilt delivered data I always get error
xcodebuild: error: The flag -testPlan cannot be used since the scheme does not use test plans.

When I don't use deriveredDataPath and test_without_building = false (it builds the app for each test) it works fine without errors.

fastlane version is 2.166.0
multi_scan version is v3.14.7

My code to build the app

multi_scan(
workspace: "MyProject.xcworkspace",
scheme: scheme,
destination: "platform=iOS Simulator,name=iPhone 8",
configuration: "Debug",
derived_data_path: "fastlane/deriveredData/#{scheme}",
build_for_testing: true
)

My code to run tests

multi_scan(
  workspace: "MyProject.xcworkspace",
  scheme: scheme,
  output_directory: "fastlane/tests/#{scheme}/#{testPlan}",
  output_types: "html",
  test_without_building: true,
  sdk: "iphonesimulator",
  fail_build: false,
  derived_data_path: "fastlane/deriveredData/#{scheme}",
  force_quit_simulator: false,
  suppress_xcode_output: false,
  device: testDevice,
  testplan: testPlan,
  clean: false,
  reset_simulator: false,
  code_coverage: false,
  disable_concurrent_testing: true,
  parallel_testrun_count: 1,
  configuration: "Debug",
  xcargs: "COMPILER_INDEX_STORE_ENABLE=NO",
  try_count: 2,
  batch_count: 3,
  testrun_completed_block: test_run_block, 
  collate_reports: false
)

Complete output when running fastlane, including the stack trace and command used

DEBUG [2020-11-04 11:28:43.83]: Fetching available simulator devices
INFO [2020-11-04 11:28:44.69]: Disabling 'Slide to Type' iPhone 8
INFO [2020-11-04 11:28:44.69]: $ /usr/libexec/PlistBuddy -c "Add :KeyboardContinuousPathEnabled bool false" /Users/asdasd/Library/Developer/CoreSimulator/Devices/71A78455-B2CA-4AAB-8688-858939E500B2/data/Library/Preferences/com.apple.keyboard.ContinuousPath.plist >/dev/null 2>&1
INFO [2020-11-04 11:28:44.72]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -sdk 'iphonesimulator' -destination 'platform=iOS Simulator,id=71A78455-B2CA-4AAB-8688-858939E500B2' -derivedDataPath fastlane/deriveredData/PROJECT-TESTS -disable-concurrent-testing -enableCodeCoverage NO -testPlan 'PROJECT-Tests' -xctestrun 'fastlane/deriveredData/PROJECT-TEST/Build/Products/PROJECT-PROJECT-Tests_iphonesimulator14.0-x86_64.xctestrun' COMPILER_INDEX_STORE_ENABLE=NO -parallel-testing-enabled NO -only-testing:BaseAppTests test-without-building | tee '/Users/asdasdsdf/Library/Logs/scan/PROJECT-TEST.log' | xcpretty --report html --output '/Users/fgdfgadfga/Documents/Projects/PROJECT-Tests/BaseAppTests-batch-1/report.html' --report junit --output '/Users/ghsfghsfgh/fastlane/tests/PROJECT-TEST/PROJECT-Tests/BaseAppTests-batch-1/report.xml' --report junit --output '/var/folders/9p/vmz4ftl13ls8ydtfygy_4tth0000gp/T/junit_report20201104-26225-17ip35w'
INFO [2020-11-04 11:28:44.72]: ▸ Loading...
INFO [2020-11-04 11:28:47.01]: ▸ xcodebuild: error: The flag -testPlan cannot be used since the scheme does not use test plans.
xcodebuild: error: The flag -testPlan cannot be used since the scheme does not use test plans.
ERROR [2020-11-04 11:28:47.03]: Exit status: 64
DEBUG [2020-11-04 11:28:47.03]: Batch #1 incrementing retry count to 1
DEBUG [2020-11-04 11:28:47.03]: Scan unable to test for batch #1
ERROR [2020-11-04 11:28:47.03]:
INFO [2020-11-04 11:28:47.03]: A everything is fine, let's continue try 1 for batch 1
WARN [2020-11-04 11:28:47.03]: Lane Context:
INFO [2020-11-04 11:28:47.03]: {:PLATFORM_NAME=>nil, :LANE_NAME=>"tests", :VERSION_NUMBER=>"6.5"}
ERROR [2020-11-04 11:28:47.03]: Error building/testing the application. See the log above.
INFO [2020-11-04 11:28:47.03]: Successfully generated documentation at path '/Users/fastlane/README.md'

bundler: failed to load command: fastlane (/usr/local/bin/fastlane)
FastlaneCore::Interface::FastlaneBuildFailure: [!] Error building/testing the application. See the log above.
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane_core/lib/fastlane_core/ui/interface.rb:163:in build_failure!' /Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane_core/lib/fastlane_core/ui/ui.rb:17:in method_missing'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/scan/lib/scan/error_handler.rb:46:in handle_build_error' /Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/scan/lib/scan/runner.rb:74:in block in test_app'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane_core/lib/fastlane_core/command_executor.rb:90:in execute' /Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/scan/lib/scan/runner.rb:65:in test_app'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/scan/lib/scan/runner.rb:23:in run' /Library/Ruby/Gems/2.6.0/gems/fastlane-plugin-test_center-3.14.7/lib/fastlane/plugin/test_center/helper/multi_scan_manager/retrying_scan.rb:77:in run'
/Library/Ruby/Gems/2.6.0/gems/fastlane-plugin-test_center-3.14.7/lib/fastlane/plugin/test_center/helper/multi_scan_manager/retrying_scan.rb:63:in run' /Library/Ruby/Gems/2.6.0/gems/fastlane-plugin-test_center-3.14.7/lib/fastlane/plugin/test_center/helper/multi_scan_manager/test_batch_worker.rb:18:in run'
/Library/Ruby/Gems/2.6.0/gems/fastlane-plugin-test_center-3.14.7/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:239:in block in run_test_batches' /Library/Ruby/Gems/2.6.0/gems/fastlane-plugin-test_center-3.14.7/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:236:in each'
/Library/Ruby/Gems/2.6.0/gems/fastlane-plugin-test_center-3.14.7/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:236:in each_with_index' /Library/Ruby/Gems/2.6.0/gems/fastlane-plugin-test_center-3.14.7/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:236:in run_test_batches'
/Library/Ruby/Gems/2.6.0/gems/fastlane-plugin-test_center-3.14.7/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:107:in block in run' /Library/Ruby/Gems/2.6.0/gems/fastlane-plugin-test_center-3.14.7/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:134:in block in setup_run_tests_for_each_device'
/Library/Ruby/Gems/2.6.0/gems/fastlane-plugin-test_center-3.14.7/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:123:in each' /Library/Ruby/Gems/2.6.0/gems/fastlane-plugin-test_center-3.14.7/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:123:in each_with_index'
/Library/Ruby/Gems/2.6.0/gems/fastlane-plugin-test_center-3.14.7/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:123:in setup_run_tests_for_each_device' /Library/Ruby/Gems/2.6.0/gems/fastlane-plugin-test_center-3.14.7/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:105:in run'
/Library/Ruby/Gems/2.6.0/gems/fastlane-plugin-test_center-3.14.7/lib/fastlane/plugin/test_center/actions/multi_scan.rb:43:in run' /Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/runner.rb:263:in block (2 levels) in execute_action'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/actions/actions_helper.rb:69:in execute_action' /Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/runner.rb:255:in block in execute_action'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/runner.rb:229:in chdir' /Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/runner.rb:229:in execute_action'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/runner.rb:157:in trigger_action_by_name' /Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/fast_file.rb:159:in method_missing'
Fastfile:95:in _run_tests' Fastfile:64:in block (2 levels) in parsing_binding'
Fastfile:63:in each' Fastfile:63:in block in parsing_binding'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/lane.rb:33:in call' /Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/runner.rb:49:in block in execute'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/runner.rb:45:in chdir' /Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/runner.rb:45:in execute'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/lane_manager.rb:47:in cruise_lane' /Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/command_line_handler.rb:36:in handle'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/commands_generator.rb:108:in block (2 levels) in run' /Users/anton_komin/.gem/ruby/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in call'
/Users/anton_komin/.gem/ruby/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in run' /Users/anton_komin/.gem/ruby/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in run_active_command'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in run!' /Users/anton_komin/.gem/ruby/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in run!'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/commands_generator.rb:352:in run' /Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/commands_generator.rb:41:in start'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/fastlane/lib/fastlane/cli_tools_distributor.rb:119:in take_off' /Library/Ruby/Gems/2.6.0/gems/fastlane-2.166.0/bin/fastlane:23:in <top (required)>'
/usr/local/bin/fastlane:23:in load' /usr/local/bin/fastlane:23:in <top (required)>'

Environment

it doesn't affect this script

Answer 1 · 2020-11-13T22:32:19.000Z

Hello @KITSEPAMAntonKomin, thank you for taking the time to report the issue. I have begun reviewing this, and would like to repeat what you wrote, in my own words, to confirm that I understand what you're saying.

When you provide pre-build artifacts, built with multi_scan and :build_for_testing, subsequent runs with :test_without_building is failing if the testplan option is provided.

Is this correct?

Answer 2 · 2020-11-13T22:34:51.000Z

Yes, correct

…

________________________________ От: Lyndsey Ferguson <notifications@github.com> Отправлено: Saturday, November 14, 2020 12:32:31 AM Кому: lyndsey-ferguson/fastlane-plugin-test_center <fastlane-plugin-test_center@noreply.github.com> Копия: Anton Komin <Anton_Komin@epam.com>; Mention <mention@noreply.github.com> Тема: Re: [lyndsey-ferguson/fastlane-plugin-test_center] multi_scan The flag -testPlan <name> cannot be used error when test_without_building = true (#310) Hello @KITSEPAMAntonKomin<https://urldefense.com/v3/__https://github.com/KITSEPAMAntonKomin__;!!GF_29dbcQIUBPA!jE9D9_0VAG1p1H2MyiOaarRlq7zqkELsAsXfSl-KlGIpcPpBYi6lUQQUwAVniX4Z$>, thank you for taking the time to report the issue. I have begun reviewing this, and would like to repeat what you wrote, in my own words, to confirm that I understand what you're saying. When you provide pre-build artifacts, built with multi_scan and :build_for_testing, subsequent runs with :test_without_building is failing if the testplan option is provided. Is this correct? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/issues/310*issuecomment-727067990__;Iw!!GF_29dbcQIUBPA!jE9D9_0VAG1p1H2MyiOaarRlq7zqkELsAsXfSl-KlGIpcPpBYi6lUQQUwPHWFsyf$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AOZEY2CKUJAP5K67Z26EJSTSPWXX7ANCNFSM4TJ4LIZA__;!!GF_29dbcQIUBPA!jE9D9_0VAG1p1H2MyiOaarRlq7zqkELsAsXfSl-KlGIpcPpBYi6lUQQUwPTWNmLF$>.

Answer 3 · 2020-11-13T23:42:21.000Z

The :testplan option should not be passed to scan as I see in the logs. The code that collects the tests should pull in the tests and populate :only_testing and the :testplan option should be rejected from the options that are being passed to scan.

This is the rejection. However, it will use the value provided by a Scanfile, so if testplan is specified there, it will have the problem. I've duplicated that and have written some code to clear that out.

Answer 4 · 2020-11-13T23:59:52.000Z

@KITSEPAMAntonKomin can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-310-testplan-passed-to-scan"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

Answer 5 · 2020-11-15T17:39:14.000Z

@lyndsey-ferguson I've tried to use your plugin version, but unfortunately it also fails with error.

DEBUG [2020-11-15 19:30:16.53]: Fetching available simulator devices
INFO [2020-11-15 19:30:17.00]: Disabling 'Slide to Type' iPhone 8
INFO [2020-11-15 19:30:17.00]: $ /usr/libexec/PlistBuddy -c "Add :KeyboardContinuousPathEnabled bool false" /Users/anton_komin/Library/Developer/CoreSimulator/Devices/71A78455-B2CA-4AAB-8688-858939E500B2/data/Library/Preferences/com.apple.keyboard.ContinuousPath.plist >/dev/null 2>&1
INFO [2020-11-15 19:30:17.00]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -sdk 'iphonesimulator' -destination 'platform=iOS Simulator,id=71A78455-B2CA-4AAB-8688-858939E500B2' -derivedDataPath fastlane/deriveredData/BASE-TEST -disable-concurrent-testing -enableCodeCoverage NO -xctestrun 'fastlane/deriveredData/BASE-TEST/Build/Products/BASE-TEST_BASE-UITests_iphonesimulator14.0-x86_64.xctestrun' COMPILER_INDEX_STORE_ENABLE=NO -parallel-testing-enabled NO -only-testing:MyProjectBaseAppTests test-without-building | tee '/Users/anton_komin/Library/Logs/scan/B&Q-BASE-TEST.log' | xcpretty --report html --output '/Users/anton_komin/Documents/Projects/MyProject/fastlane/tests/BASE-TEST/BASE-Tests/MyProjectBaseAppTests-batch-1/report.html' --report junit --output '/Users/anton_komin/Documents/Projects/MyProject/fastlane/tests/BASE-TEST/BASE-Tests/MyProjectBaseAppTests-batch-1/report.xml' --report junit --output '/var/folders/9p/vmz4ftl13ls8ydtfygy_4tth0000gp/T/junit_report20201115-23908-9c45ut'
INFO [2020-11-15 19:30:17.00]: ▸ Loading...
INFO [2020-11-15 19:30:19.90]: ▸ xcodebuild: error: Failed to build workspace temporary with scheme Transient Testing.
INFO [2020-11-15 19:30:19.90]: ▸ Reason: Tests in the target “MyProjectBaseAppTests” can’t be run because “MyProjectBaseAppTests” isn’t a member of the specified scheme.
xcodebuild: error: Failed to build workspace temporary with scheme Transient Testing.
Reason: Tests in the target “MyProjectBaseAppTests” can’t be run because “MyProjectBaseAppTests” isn’t a member of the specified scheme.
ERROR [2020-11-15 19:30:19.92]: Exit status: 70
DEBUG [2020-11-15 19:30:19.92]: Batch #1 incrementing retry count to 1
DEBUG [2020-11-15 19:30:19.92]: Scan unable to test for batch #1
ERROR [2020-11-15 19:30:19.92]:
INFO [2020-11-15 19:30:19.92]: A everything is fine, let's continue try 1 for batch 1
WARN [2020-11-15 19:30:19.92]: Lane Context:
INFO [2020-11-15 19:30:19.92]: {:PLATFORM_NAME=>nil, :LANE_NAME=>"tests", :VERSION_NUMBER=>"6.5"}
ERROR [2020-11-15 19:30:19.92]: Error building/testing the application. See the log above.
INFO [2020-11-15 19:30:19.92]: Successfully generated documentation at path '/Users/anton_komin/Documents/Projects/MyProject/fastlane/README.md'

+------+---------------------------------------------------------------+-------------+
| fastlane summary |
+------+---------------------------------------------------------------+-------------+
| Step | Action | Time (in s) |
+------+---------------------------------------------------------------+-------------+
| 1 | Verifying fastlane version | 0 |
| 2 | update_project_team | 1 |
| 3 | set_info_plist_value | 0 |
| 4 | get_version_number | 0 |
| 5 | Config/brand.sh BASE-TEST /Users/anton_komin/Documents/Projec | 0 |
| 6 | /Users/anton_komin/Documents/Projects/MyProject | 0 |
| 💥 | multi_scan | 16 |
+------+---------------------------------------------------------------+-------------+

DEBUG [2020-11-15 19:30:19.93]: All plugins are up to date
ERROR [2020-11-15 19:30:19.93]: fastlane finished with errors
DEBUG [2020-11-15 19:30:19.93]: All plugins are up to date
`

Answer 6 · 2020-11-16T13:40:16.000Z

Reviewing this error:

xcodebuild: error: Failed to build workspace temporary with scheme Transient Testing.
INFO [2020-11-15 19:30:19.90]: ▸ Reason: Tests in the target “MyProjectBaseAppTests” can’t be run because “MyProjectBaseAppTests” isn’t a member of the specified scheme.
xcodebuild: error: Failed to build workspace temporary with scheme Transient Testing.
Reason: Tests in the target “MyProjectBaseAppTests” can’t be run because “MyProjectBaseAppTests” isn’t a member of the specified scheme.

Makes me suspect that the test target MyProjectBaseAppTests is not included in the BASE-TEST Xcode Scheme. Can you double check that it is included?

Answer 7 · 2020-11-16T13:58:12.000Z

@lyndsey-ferguson
Yes, indeed it is included.

I'm getting this when I try to launch tests from build deriveredData - "No selected tests, using testable 'MyProjectBaseAppTests'"
This log goes before one that I've already added in previous message

INFO [2020-11-15 19:30:07.32]: $ xcodebuild -showBuildSettings -workspace MyProject.xcworkspace -scheme MyProject-TEST -configuration Debug -derivedDataPath fastlane/deriveredData/MyProject-TEST WARN [2020-11-15 19:30:10.32]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout... DEBUG [2020-11-15 19:30:14.13]: Fetching available simulator devices DEBUG [2020-11-15 19:30:14.33]: Looking in Scheme '/Users/anton_komin/Documents/Projects/Project-gitlab/Project/MyProjectBaseApp.xcodeproj/xcshareddata/xcschemes/MyProject-TEST.xcscheme' for any testplans DEBUG [2020-11-15 19:30:14.36]: found testplan '/Users/anton_komin/Documents/Projects/Project-gitlab/Project/TestPlans/MyProject/MyProject-TESTs.xctestplan' DEBUG [2020-11-15 19:30:14.36]: found testplan '/Users/anton_komin/Documents/Projects/Project-gitlab/Project/TestPlans/MyProject/MyProject-SnapshotTests.xctestplan' DEBUG [2020-11-15 19:30:14.36]: found testplan '/Users/anton_komin/Documents/Projects/Project-gitlab/Project/TestPlans/MyProject/MyProject-UITests.xctestplan' DEBUG [2020-11-15 19:30:14.36]: TestCollector found testplans: ["/Users/anton_komin/Documents/Projects/Project-gitlab/Project/TestPlans/MyProject/MyProject-TESTs.xctestplan", "/Users/anton_komin/Documents/Projects/Project-gitlab/Project/TestPlans/MyProject/MyProject-SnapshotTests.xctestplan", "/Users/anton_komin/Documents/Projects/Project-gitlab/Project/TestPlans/MyProject/MyProject-UITests.xctestplan"] DEBUG [2020-11-15 19:30:14.36]: using :testplan option, MyProject-TESTs, using found one: /Users/anton_komin/Documents/Projects/Project-gitlab/Project/TestPlans/MyProject/MyProject-TESTs.xctestplan DEBUG [2020-11-15 19:30:14.36]: Examining testplan JSON: {"configurations"=>[{"id"=>"8FEA8EF6-BA1A-4540-B858-E19B9B5F8B4D", "name"=>"Configuration 1", "options"=>{}}], "defaultOptions"=>{"environmentVariableEntries"=>[{"key"=>"IMAGE_DIFF_DIR", "value"=>"$(SOURCE_ROOT)/$(PROJECT_NAME)SnapshotTests/SnapshotImages/MyProject/FailureDiffs"}, {"key"=>"FB_REFERENCE_IMAGE_DIR", "value"=>"$(SOURCE_ROOT)/$(PROJECT_NAME)SnapshotTests/SnapshotImages/MyProject/ReferenceImages"}], "mainThreadCheckerEnabled"=>false, "targetForVariableExpansion"=>{"containerPath"=>"container:MyProjectBaseApp.xcodeproj", "identifier"=>"4F4891F61DCB84D300E74ECD", "name"=>"MyProjectBaseApp"}}, "testTargets"=>[{"target"=>{"containerPath"=>"container:MyProjectBaseApp.xcodeproj", "identifier"=>"4F48920A1DCB84D300E74ECD", "name"=>"MyProjectBaseAppTests"}}, {"target"=>{"containerPath"=>"container:PlacesAutocomplete/PlacesAutocomplete.xcodeproj", "identifier"=>"073D18CC1FC43F4B0054AA43", "name"=>"PlacesAutocompleteTests"}}, {"target"=>{"containerPath"=>"container:AppConfiguration/AppConfiguration.xcodeproj", "identifier"=>"0764AD941FCECAA600270BDF", "name"=>"AppConfigurationTests"}}, {"target"=>{"containerPath"=>"container:Config/OpCo/OpCoDependencies.xcodeproj", "identifier"=>"950199D9FB0C61E7A9ABC3B0", "name"=>"OpCoDependencyTests"}}, {"target"=>{"containerPath"=>"container:MyProjectBaseApp.xcodeproj", "identifier"=>"9617BF6C21F20D3400AB8976", "name"=>"MyProjectBaseAppSnapshotTests"}}, {"target"=>{"containerPath"=>"container:ComponentLibrary/ComponentLibrary.xcodeproj", "identifier"=>"076598CC22C4BE8D00ADD39C", "name"=>"ComponentLibraryTests"}}, {"target"=>{"containerPath"=>"container:Modules/MyProjectModel/MyProjectModel.xcodeproj", "identifier"=>"BEA6510324890547005E57C8", "name"=>"MyProjectModelTests"}}, {"target"=>{"containerPath"=>"container:Modules/Loyalty/ClubCard/ClubCard.xcodeproj", "identifier"=>"9D08A2B3212AEB8E00FDB8E6", "name"=>"ClubCardTests"}}, {"target"=>{"containerPath"=>"container:Modules/Loyalty/Loyalty/Loyalty.xcodeproj", "identifier"=>"96C4EEC9227B39D10090C26D", "name"=>"LoyaltyTests"}}], "version"=>1} DEBUG [2020-11-15 19:30:14.36]: No selected tests, using testable 'MyProjectBaseAppTests' DEBUG [2020-11-15 19:30:14.36]: No selected tests, using testable 'AppConfigurationTests' DEBUG [2020-11-15 19:30:14.36]: No selected tests, using testable 'MyProjectBaseAppSnapshotTests' DEBUG [2020-11-15 19:30:14.36]: No selected tests, using testable 'MyProjectModelTests' DEBUG [2020-11-15 19:30:14.36]: Getting tests from xctestrun file at 'fastlane/deriveredData/MyProject-TEST/Build/Products/MyProject-TEST_MyProject-UITests_iphonesimulator14.0-x86_64.xctestrun'

Answer 8 · 2020-11-18T14:08:51.000Z

@KITSEPAMAntonKomin it looks like the action meant to parse the testplan, test_options_from_testplan is unable to parse the given test plan. It is looking for the key "selectedTests" in each test target. My guess is that it is supposed to use the "name" field in each test target in this particular case.

Are the values "MyProjectBaseAppSnapshotTests", "ComponentLibraryTests", "MyProjectModelTests", "ClubCardTests", "LoyaltyTests" test suite classes in the targets "MyProjectBaseApp", "ComponentLibrary", "MyProjectModel", "ClubCard", and "Loyalty" respectively?

Answer 9 · 2020-11-19T08:44:21.000Z

@lyndsey-ferguson
I've checked it and yes, test suite classes are added to their test targets as members.
If I use scan with testPlan parameter it works fine (with or without test_without_building = true), but we have some flaky tests and that is why we need functionality of multi_scan to retry them...

Answer 10 · 2020-11-19T14:36:12.000Z

Ok, great, thank you for the information and for collaborating with me on this issue.

I will have to update the test_options_from_testplan action to better parse this information. Would you be able to upload a copy of the testplan document? You can change the names of things if you want, but I'd like the full format in order to iterate on fixing my code.

Once I have something that I think will work, I'll update this branch and ask you to test again, does that work for you?

Answer 11 · 2020-11-20T11:30:02.000Z

@lyndsey-ferguson
Could you give me your email ? I will send it there.

Answer 12 · 2020-11-20T13:31:02.000Z

I don't want to post my email publicly. Can you encrypt it with my public key, attached the encrypted file, and I can decrypt it on my local?

openssl rsautl -encrypt -pubin -inkey communication-public.key -in <path/to/your/xctestplan> -out encrypted.xctestplan

communication-public.key.zip

Answer 13 · 2020-11-20T13:43:19.000Z

@lyndsey-ferguson ok, good idea with encryption, but data size of 470 kb is too large for your public key (I tried to encrypt and got this error - "data too large for key size")

Answer 14 · 2020-11-20T13:50:35.000Z

Okay, I'll set up another key with GPG later on.

Answer 15 · 2020-11-21T14:57:23.000Z

GPG is no longer working on macOS 11. So, I used this article to guide me on how we can exchange this file.

Here's the short version.

Create a password file that you will use to encrypt your testplan file:

openssl rand 192 -out secret.txt.key

Encrypt the testplan file with that secret:

$ openssl aes-256-cbc -in <path/to/your/xctestplan> -out <path/to/your/xctestplan>.enc -pass file:secret.txt.key

Encrypt the password file with the attached public key:

openssl rsautl -encrypt -pubin -inkey lyndsey-ferguson-id_rsa.pub.pkcs8 -in secret.txt.key -out secret.txt.key.enc

Package up the encrypted files:

zip issue-310.zip *.enc

Attach that zip to this issue.

lyndsey-ferguson-id_rsa.pub.pkcs8.zip

Answer 16 · 2020-11-25T13:37:11.000Z

MyProject-testplan.zip
@lyndsey-ferguson sorry for the delay

Answer 17 · 2020-11-29T15:53:22.000Z

Ok, I don't know why I had to do do this, but I had to use a docker image with openssl to successfully decode this test plan. I think it is something to do with the latest OpenSSL on the new macOS.

Answer 18 · 2020-11-29T16:31:00.000Z

I created this lane:

lane :issue_310 do
  test_options = test_options_from_testplan(
    testplan: 'MyProject-TESTS.xctestplan'
  )
  only_testing = test_options[:only_testing]
  only_testing.each do |test_identifier|
    puts test_identifier
  end
end

And ran it on the testplan that you provided (Thanks!).

However, it seems to be parsing the testplan file correctly. With the --verbose flag, I do not see this output (from the above comment indicating a problem):

DEBUG [2020-11-15 19:30:14.36]: No selected tests, using testable 'MyProjectBaseAppTests'
DEBUG [2020-11-15 19:30:14.36]: No selected tests, using testable 'AppConfigurationTests' 
DEBUG [2020-11-15 19:30:14.36]: No selected tests, using testable 'MyProjectBaseAppSnapshotTests' 
DEBUG [2020-11-15 19:30:14.36]: No selected tests, using testable 'MyProjectModelTests'

Can you copy the lane that I created and run it as well?

bundle exec fastlane issue_310 --verbose

When I ran it, I got 9797 lines of text from the point that it printed out:

   1 DEBUG [2020-11-29 11:31:51.88]:   Found selectedTests

And based on what I saw in the .testplan file and the output, it is finding all the tests.

Answer 19 · 2020-11-29T16:36:06.000Z

My other thought is that the logic that is pulling the tests for the only_testing is failing. Looking at that.

Answer 20 · 2020-11-29T17:02:20.000Z

Okay, I don't see anything, but I have added more logging to this git branch.

@KITSEPAMAntonKomin can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-310-testplan-passed-to-scan"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know. I want to see how many tests are being sent to the underlying scan action. My testing indicates that it should be sending each test target's tests to scan in a separate run. My guess is that the first "batch" of tests has an only_testing that is just "MyProjectBaseAppTests". All the same, this shouldn't lead to the problem that you're seeing.

You write that the Scheme does have that test target. Can you take a picture of the Scheme with the different sections (as per my example images). If you need to encrypt them and attach them with my public key, that's fine.

Answer 21 · 2020-11-29T17:12:32.000Z

So, @KITSEPAMAntonKomin , you have two assignments 😄

see this comment
see this comment

Answer 22 · 2020-11-29T17:23:43.000Z

It may get to the point where I have to ask for a sample project to properly debug this.

Answer 23 · 2020-11-30T21:37:53.000Z

Hello @lyndsey-ferguson, I am also trying to use an .xctestplan file with some defined tests to run in our current project.

I just saw this issue now, I went ahead and created a new empty project with one Test target, one class of Test with 3 tests, and one .xctestplan file with one selected test.
I updated the Pluginfile per your instruction above and created the issue_310 lane.

However when running the lane I get the following: No selected tests, using testable 'FastlaneTestPlanUnitTests'

Looking at the .xctestplan file it only mention the skipped tests not the selected tests, is that the parsing issue here?

I'm attaching this sample project here.

Let me know if you need anything else and thank you very much for looking into this!

Answer 24 · 2020-12-03T13:42:27.000Z

@adrienyvon699 your sample project is useful in demonstrating a bug, but I don't think it is the same bug. All the same, I'll update this branch with some work.

Oh, and the bug is: your test plan has skipped tests, and the test_options_from_testplan action is not parsing them.

Answer 25 · 2020-12-03T15:16:31.000Z

@lyndsey-ferguson
Files of 2 assignments )
Encrypted as my previous files
Logs and screens.zip

Answer 26 · 2020-12-03T18:04:24.000Z

@adrienyvon699 your sample project is useful in demonstrating a bug, but I don't think it is the same bug. All the same, I'll update this branch with some work.

Oh, and the bug is: your test plan has skipped tests, and the test_options_from_testplan action is not parsing them.

Yes that sounds right! Thank you again for looking into it, I'll be watching the thread.

Answer 27 · 2020-12-12T14:30:33.000Z

@adrienyvon699 I believe that I've fixed the skipped_tests issue, could you remove your Gemfile.lock file, make sure that you've modified your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-310-testplan-passed-to-scan"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

Answer 28 · 2020-12-12T14:31:15.000Z

@KITSEPAMAntonKomin downloading your sample project now!
(apologies for slow responses -- I'm super busy and have limited time to work on this plugin)

Answer 29 · 2020-12-12T14:54:46.000Z

I apologize, I am about to run out of the time I can spend on this plugin for now. I was able to successfully decrypt and unzip the screenshots and review them. I haven't had enough time to really analyze them, and so much time has passed since we last communicated that I need to re-read the thread of this Issue to figure out why I asked for the screenshots. I see that you have test plans in them, but I cannot recall what I was hoping to get out of them.

I also do not see any errors in the logs you sent. Based on the empty lines, I would guess that you've redacted a lot. I do see that the run of fastlane runs successfully, so I'm not even sure that there is an error anymore! 😊

@KITSEPAMAntonKomin can you help me out and provide a summary of the current problem. Apologies. I will try and at least write out the next steps so I can recapture the context.

Answer 30 · 2020-12-12T14:56:47.000Z

For myself, next time:

Review this Issue thread to understand why I was asking for screenshots.
Analyze the provided screenshots to see if the bug submitter has configured his project in an unanticipated way.
Based on the information that the bug submitter provides, review the logs for why he/she may be experiencing an issue.

Answer 31 · 2020-12-16T14:41:44.000Z

@lyndsey-ferguson
Current issue is that we can't run unit tests without building (from precompiled delivered data, build for testing = true). multi_scan returns error "xcodebuild: error: The flag -testPlan cannot be used since the scheme does not use test plans."
We want to have set of different test plans for specific test scenarios and to run them after only 1 project compilation instead of test with compilation for each test plan which dramatically increases CI pipelines time

Answer 32 · 2020-12-16T16:45:14.000Z

@lyndsey-ferguson
I've tried this version :branch => "issue-310-testplan-passed-to-scan"

and got this error

INFO [2020-12-16 18:39:51.36]: ▸ xcodebuild: error: Failed to build workspace temporary with scheme Transient Testing.
INFO [2020-12-16 18:39:51.36]: ▸ Reason: Tests in the target “MyAppBaseAppTests” can’t be run because “MyAppBaseAppTests” isn’t a member of the specified scheme.

But it correctly found all my tests inside passed test plan.

Answer 33 · 2021-01-17T15:30:14.000Z

Reviewing this thread, I've caught up and the current state is that the project of @KITSEPAMAntonKomin cannot test pre-built data because 1. the testplan option is being provided to scan in appropriately, 2. when I fix that issue, we get an error with xcodebuild unable to "build workspace temporary with scheme transient testing"

Answer 34 · 2021-01-17T15:31:01.000Z

Next up: analyze the provided screenshots to look for anything unanticipated.
Then: analyze logs for any clues.

Answer 35 · 2021-02-11T20:29:32.000Z

@adrienyvon699 I believe that I've fixed the skipped_tests issue, could you remove your Gemfile.lock file, make sure that you've modified your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:
gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-310-testplan-passed-to-scan"
If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

Hi @lyndsey-ferguson I am so sorry for being so late in responding to you. I understand that it's been a while and it might be hard to jump back into this.
I ran it again per your instructions and seems to have the same issue:

INFO [2021-02-11 15:25:01.56]: ------------------------------
INFO [2021-02-11 15:25:01.56]: --- Step: default_platform ---
INFO [2021-02-11 15:25:01.56]: ------------------------------
INFO [2021-02-11 15:25:01.56]: Driving the lane 'ios issue_310' 🚀
INFO [2021-02-11 15:25:01.56]: ----------------------------------------
INFO [2021-02-11 15:25:01.56]: --- Step: test_options_from_testplan ---
INFO [2021-02-11 15:25:01.56]: ----------------------------------------
DEBUG [2021-02-11 15:25:01.56]: Examining testplan JSON: {"configurations"=>[{"id"=>"A81A89CF-FC8B-4D87-A4AD-1052C2E5A6A1", "name"=>"Configuration 1", "options"=>{}}], "defaultOptions"=>{}, "testTargets"=>[{"skippedTests"=>["FastlaneTestPlanUnitTests/test2()", "FastlaneTestPlanUnitTests/test3()"], "target"=>{"containerPath"=>"container:FastlaneTestPlan.xcodeproj", "identifier"=>"E64A7F9225759881003523F6", "name"=>"FastlaneTestPlanUnitTests"}}], "version"=>1}
DEBUG [2021-02-11 15:25:01.57]:   No selected tests, using testable 'FastlaneTestPlanUnitTests'```

Answer 36 · 2021-03-02T13:12:50.000Z

I don't see anything suspicious with the images of the Xcode Scheme configuration. I do see that Anton is using test plans in the test stage.

Answer 37 · 2021-03-02T13:19:08.000Z

@KITSEPAMAntonKomin can you provide a sample project that exhibits this problem? I'm afraid I need it to proceed further.

Answer 38 · 2021-03-22T17:36:30.000Z

Closing this ticket as I have no way to reproduce and fix this issue.