invalid option '-parallel-testing-enabled' on older Xcode versions

New Issue Checklist

Updated fastlane-plugin-test_center to the latest version
I read the README.md
I reviewed the example(s) for the action(s) I am using
I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see

If you love this fastlane plugin, consider sponsoring it or asking your company to sponsor it. I would really appreciate any
gesture: https://github.com/sponsors/lyndsey-ferguson. 😍

Issue Description

Xcode 9.2 fails (newest on macOS 10.12) with this problem, Xcode 10.1 succeeds (newest on macOS 10.13), cutoff is probably 10.0

fastlane-plugin-test_center/lib/fastlane/plugin/test_center/helper/multi_scan_manager/retrying_scan_helper.rb

Line 105 in 4080a22

xcargs: "#{xcargs} -parallel-testing-enabled NO "

Fastlane checks for Xcode 10 before setting similar options:
https://github.com/fastlane/fastlane/blob/5d9219e34f21e6e3bb87df87a5e621f1e7e42edc/scan/lib/scan/test_command_generator.rb#L42

It might be also worth implementing a supported options check by parsing the available options from the output of xcodebuild --help and removing the option before running and printing a warning.

@Cyberbeni, can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-305-turn-off-parallel-only-for-xcode10-or-greater"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

It gives a different error now. Compared to the currently working version, it passes the xctestrun parameter instead of the workspace and scheme parameters into test-without-building (The xctestrun file is the one built with the newest Xcode, the whole derived data is transferred to machine with old macOS after build-for-testing, project file is modified, so build-for-testing the test target doesn't rebuild the app, only the test runner and then the build products are copied over to the transferred derived data folder after deleting the test runner part of the old build products)
10.11 test error.txt

So my guess that this also needs an Xcode version check:

fastlane-plugin-test_center/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb

Line 230 in c8526f8

pool_options[:xctestrun] = @test_collector.xctestrun_path

I tested it and modifying the linked line to check for xcode version solves the issue

          if FastlaneCore::Helper.xcode_at_least?(10)
            pool_options[:xctestrun] = @test_collector.xctestrun_path
          end

(Also I know that the output contains color codes, that's a different issue: fastlane/fastlane#17498 )

Hi @Cyberbeni it does not feel that this is a generalizable solution. From what I recall, I've always needed a xctestrun when running test-without-building. I believe that xcodebuild can get the xctestrun file from the derived data file path though.

Is there no xctestrun file in the derived data folder that was copied over from the older macOS build?

From what I remember, the only difference in xctestrun files was from removing the target application before rebuilding the test runner, so I'm keeping the xctestrun file built with newer Xcode.

Which is correctly found according to the log.

Ah, I totally forgot about the actual log file, that might tell us more about the issue than the xcpretty formatted version. I'll take a look at it on Monday.

Ah, I guess that wasn't the only difference between the xctestrun files: dyld: could not load inserted library '__DEVELOPERUSRLIB__/libMainThreadChecker.dylib' because image not found
But it would still be nice to not have to hack around with merging the 2 xctestrun files (That would just add one more possible point of failures) when using the project+scheme is perfectly fine.

I was thinking that it should be possible to use the old xctestrun file 🤞

The best solution might be to add an option to skip setting the xctestrun file as temporarily downgrading the project file for running tests is a supported use case by Xcode.

@Cyberbeni let me walk through what I understand the situation you find yourself in.

On a Mac, you're using Xcode 9.2. The build (and the derived data folder) were created with Xcode 9.2. Your test job downloads the build and the associated derived data folder and your scan (and multi_scan) paths point appropriately to those directories.

Request: Can you provide an example of the options that you are passing to scan and multi_scan?

scan will work fine: it finds what it needs and runs the tests.

multi_scan, on the other hand, fails. It is using an xctestrun file generated by a newer Xcode, correct? Do you have information on why this newer xctestrun file is being picked up? Should it not be using the older one? Where is this newer xctestrun file stored? I have a class, TestCollector that uses either the provided :xctestrun field, or looks in the provided :derived_data_path, or failing that, it looks in the Project's path for where it would put the :derived_data_path.

The reason that I hesitate making your suggested change is that it would break the mechanism that multi_scan is able to determine the tests to test (given partial test identifiers: i.e. -only-testing:TestTarget/TestSuite/testCase).

First machine has newest MacOS and newest Xcode and runs xcodebuild build-for-testing with the appropriate commands.

Second machine has macOS 10.11 running Xcode 8.2 (and other macOS versions with the newest Xcode that supports them), downloads the code and the prebuilt derived data. Modifies the project file by removing the target application from the test target, so it can run xcodebuild build-for-testing without failing on the app containing Swift 5.0+ code and taking more time for unnecessary builds (it also does some other minor modifications that are irrelevant in this case) then it copies the built products from Xcode 8.2 and overwrites the ones built with the newest Xcode then executes test-without-building specifying the path for this merged DerivedData folder.

The current version of multi_scan being used and running tests properly is from before adding parallel testing on iOS. The xctestrun file in the derived data is 100% valid for determining what test cases there are but it is not suitable to pass to xcodebuild test-without-building instead of passing the workspace and scheme.

Ok, so the xctestrun is actually built by the newer machine.

So, I'm going to allow my consciousness to stream, to generate ideas. Maybe some of them are good, maybe some are trash.

Right now, the problem is that the version of the xctestrun file is not really compatible with the older version of Xcode.

I've reviewed the file that was attached in this comment. It indicates that for that time, it was crashing because:

INFO [2020-10-26 08:17:25.75]: ▸ �[0;35;49mDetails:  Unable to create attachments directory /Users/****/****/workspace/Mac-Run-Built-TestSuite/DerivedData/Logs/Test/Attachments: Error Domain=NSCocoaErrorDomain Code=516 "The file “Attachments” couldn’t be saved in the folder “Test” because a file with the same name already exists." UserInfo={NSFilePath=/Users/****/****/workspace/Mac-Run-Built-TestSuite/DerivedData/Logs/Test/Attachments, NSUnderlyingError=0x7ffec46967e0 {Error Domain=NSPOSIXErrorDomain Code=17 "File exists"}}�[0m

That may be happening because @Cyberbeni is passing in a pre-built derived data directory that contains everything, including those logs. Now, I could, for debugging purposes, add code that pre-deletes the /Users/****/****/workspace/Mac-Run-Built-TestSuite/DerivedData/Logs directory, but that seems awfully presumptuous: the user should be providing only what is needed for the tests (/Users/****/****/workspace/Mac-Run-Built-TestSuite/DerivedData/Build/Products).

However, that is not is the problem being discussed, apparently there is some indication that the xctestrun file is causing a problem (given that it is a newer version than what Xcode can read, I'm not surprised). So what is that and how can we deal with it?

First, let's examine the log again to see if we can see the error: dyld: could not load inserted library '__DEVELOPERUSRLIB__/libMainThreadChecker.dylib' because image not found. That error is not in the logs that were given.

@Cyberbeni can you provide the logs that demonstrate that error?

Let's assume for the moment that he is completely correct. What to do? Three things come immediately to mind:

Convert the newer xctestrun file to the older version. Ug, that means I have to really understand the differences. Well, I have written code to read both versions for what multi_scan needs, so maybe this is a possibility? If only I had access to a sample project and and older Mac with Xcode 10.11. If I had parallels I could emulate the older Mac setup....Hm. It's $80 USD. Let's skip that for now.
Build in a new multi_scan option that allows users to tell multi_scan, "hey, it's okay, don't pass in the xctestrun parameter, I know what I'm doing". I'm not a fan of that, but maybe there are reasons like this one that it can be make sense.
Build in a new multi_scan option that provides users a callback with the proposed settings that will be sent to scan and allow them to modify the settings. This would be for more advanced users like Cyberbeni. I like this more than no. 1 and no. 2.

Okay, I've written my thoughts so far, I'm going to let it stew a little and then maybe choose one today or tomorrow to work on.

ln -s /dev/null "$DERIVED_DATA_DIR/Logs/Test/Attachments"

Xcode 8.2 is generating those attachments to 3 different places and doesn't clean them up after a test case succeeds and we want to avoid running out of disk space while running the tests (They take around 3-4 hours currently). That is just a warning, doesn't fail the tests.

set -o pipefail && env NSUnbufferedIO=YES xcodebuild -destination 'platform=OS X,arch=x86_64' -derivedDataPath /Users/****/****/workspace/Mac-Run-Built-TestSuite/DerivedData -resultBundlePath '/Users/****/****/workspace/Mac-Run-Built-TestSuite/out/MyAppMac.test_result' -xctestrun '/Users/****/****/workspace/Mac-Run-Built-TestSuite/DerivedData/Build/Products/MyAppMac_macosx10.15-x86_64.xctestrun'  -only-testing:testNames test-without-building | tee '/Users/****/****/workspace/Mac-Run-Built-TestSuite/out/MyApp-MyAppMac.log' | xcpretty --no-color --report junit --output '/Users/****/****/workspace/Mac-Run-Built-TestSuite/out/result.0.xml' --report junit --output '/var/folders/31/lx_rvf757kbfj145_4t8nl9w0000gp/T/junit_report20201026-2861-110fz6u'

The dylib loading error was in the log file that tee generated.
Maybe the xcpretty output was filtered by this part of our workflow(But I think xcpretty just ignores that line):

		| grep -v -e "/com.apple.dt.XCTest/" \
		-e "Please file a bug" \
		-e "IDESchemeActionTestActivitySummary" \
		-e "withAttachmentsExtractedToDirectory" \
		-e "{number = 1, name = main}" \
		-e "▸ Copy" \
		-e "▸ Link" \
		-e "^Copy$" \
		-e "^Link$"
		# ^^ have to filter output to suppress 200MB of error messages due to revoking write access to Attachments directory

I think VirtualBox would be sufficient for this (haven't tried it, we are using ESXi) following this guide and getting the image from here.

dylib crash.log

If you think I missed anything please let me know.

ln -s /dev/null "$DERIVED_DATA_DIR/Logs/Test/Attachments"

Xcode 8.2 is generating those attachments to 3 different places and doesn't clean them up after a test case succeeds and we want to avoid running out of disk space while running the tests (They take around 3-4 hours currently). That is just a warning, doesn't fail the tests.

It looked like valid failure. I've seen the same kind of error when xcodebuild tries to run and a directory it wants to write to already exist. That was when I was building out the parallel runner code -- I think it was for the xcresult file directory.

But, you're saying that Xcode 8.3 generates those and is failing? Does it not fail for the first run?

Regarding VirtualBox, I always have problems when trying to set one up.

But, you're saying that Xcode 8.3 generates those and is failing? Does it not fail for the first run?

No, Xcode is not failing, it only causes warning logs during run. If we would let Xcode write to those folders, that would fill up the disk and that would make Xcode fail.

@Cyberbeni I've created a new option for multi_scan that will let you change the options that scan will receive.

Here is an example of how I could see you using it:

  testrun_prerun_block = lambda do |scan_options|
    scan_options.delete(:xctestrun)
  end

  multi_scan(
    workspace: File.absolute_path('../AtomicBoy/AtomicBoy.xcworkspace'),
    scheme: 'AtomicBoy',
    fail_build: false,
    try_count: 2,
    only_testing: ['AtomicBoyUITests', 'AtomicBoyTests'],
    testrun_prerun_block: testrun_prerun_block
  )

Can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-305-turn-off-parallel-only-for-xcode10-or-greater"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

Also, I'm not in love with the option name, if you have some better ideas, I'm open to hear them.

It works.

override_scan_options_block?

Released in Sponsors-first version of v3.15.1. Available to the general public around Dec 18th, 2020!

Any individual supporter at the Basic Supporter tier or higher will get early access, and all members of an organization sponsor at the Silver tier or higher will get that same access.

Released in v3.15.0