lyndsey-ferguson/fastlane-plugin-test_center

multi-scan parallel run with 1+ devices will only run on one of those devices

Closed this issue · 9 comments

bpoplauschi commented

New Issue Checklist

  • Updated fastlane-plugin-test_center to the latest version
  • I read the README.md
  • I reviewed the example(s) for the action(s) I am using
  • I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see

Issue Description

The issue is this:

  • parallel running multi_scan with several devices doesn't function as expected
    • if we run multi_scan with parallel off (parallel_testrun_count = 1 and batch_count = 1) multiple devices and multiple targets, then fastlane builds each target once and then runs it on every device
    • but if we enable parallelization with the same settings, multi_scan only runs on one of the devices:
  multi_scan(
    scheme: scheme, 
    devices: ["iPhone 8 (13.3)", "iPhone 8 (12.4)", "iPhone 8 (11.4)"],
    code_coverage: false, 
    derived_data_path: "./DerivedData", 
    output_directory: "./reports", 
    formatter: "xcpretty-json-formatter", 
    try_count: @retries_count,
    only_testing: ["TestsTarget1", "TestTarget2", "TestsTarget3"], 
    testrun_completed_block: test_run_block, 
    parallel_testrun_count: 3, 
    batch_count: 3
  )
  • in this case, all the targets will be tested against "iPhone 8 (13.3)".

Do you have any idea why this happens or can use any more info from me?

Environment





Key
Value






OS
10.15.4




Ruby
2.6.3




Bundler?
true




Git
git version 2.21.1 (Apple Git-122.3)




Installation Source
~/.rvm/gems/ruby-2.6.3/bin/fastlane




Host
Mac OS X 10.15.4 (19E287)




Ruby Lib Dir
~/.rvm/rubies/ruby-2.6.3/lib




OpenSSL Version
OpenSSL 1.1.1d  10 Sep 2019




Is contained
false




Is homebrew
false




Is installed via Fabric.app
false




Xcode Path
/Applications/Xcode.app/Contents/Developer/




Xcode Version
11.3.1










Gem
Version
Update-Status






fastlane
2.144.0
🚫 Update available










Plugin
Version
Update-Status






fastlane-plugin-test_center
3.10.1
🚫 Update available




fastlane-plugin-firebase_app_distribution
0.1.4
✅ Up-To-Date

PS: Very nice work, this project is very helpful! Keep up the good and constant work!

lyndsey-ferguson commented

Hello @bpoplauschi, this seems like a bug to me. It should create 3 clones of each of the devices that you provided and split the tests on each device. I don't have a lot of time to look at it right now, but I would hazard a guess that what is happening is that I am not providing the :destination correctly (only specifying one simulator) where the clone simulators are being set up

If you'd like to take a crack at it, I'm open to PRs can provide guidance. It may a while before I can get to this.

lyndsey-ferguson commented

@bpoplauschi
Can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-216-multiple-devices-doesnt-work"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

bpoplauschi commented

@lyndsey-ferguson I really appreciate you looking into this issue and fixing it. I no longer work on that project, so I can't try out the fix, but maybe one of my former teammates can look into this: @nfagen1 @jleyrer @akoller-vs (Hi guys!).

  • ❤️1
  • 👀1
lyndsey-ferguson commented

@BohdanOrlov thank you for connecting me with your former team members -- hopefully when they test, they will see that it is fixed.

I hope that you're enjoying your current project!

  • 👍1
akoller-vs commented

@lyndsey-ferguson We finally got around to testing this newer version of the plugin and parallel runs with multiple devices appears to be fixed. However, we're seeing that failed tests are only reran when they fail on all selected devices. If a test fails on only 1 device when using multiple devices, a retry test run correctly starts but doesn't actually rerun the failed tests.

  • 👍1
lyndsey-ferguson commented

@akoller-vs thanks for testing and the info! Would you be able to open up a new issue with any logs/env details that you're allowed to provide?

🙏

akoller-vs commented

@lyndsey-ferguson Sure, here it is #263.

lyndsey-ferguson commented

Thanks, prioritized!