lyndsey-ferguson/fastlane-plugin-test_center

HtmlTestReport::TestSuite::remove_duplicate_testcases can throw exception in some cases

Closed this issue · 5 comments

lilyball commented

New Issue Checklist

  • Updated fastlane-plugin-test_center to the latest version (I verified the code though)
  • I read the README.md
  • I reviewed the example(s) for the action(s) I am using
  • I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see

Issue Description

We have an intermittent error when running tests and it looks like it's caused by a bug in the code that collates html reports. In particular, TestCenter::Helper::HtmlTestReport::TestSuite::remove_duplicate_testcases has code that looks like

(nonuniq_testcases - uniq_testcases).each do |tc|
  failure_details = tc.failure_details
  tc.root.parent.delete_element(failure_details)
  tc.root.parent.delete_element(tc.root)
end

The problem is failure_details may be nil, which means it's calling tc.root.parent.delete_element(nil). The REXML documentation doesn't state what happens when passing nil, but experimentally, it deletes the first element. As such, if tc.root happens to be the first element of tc.root.parent, then this rips tc.root out of the tree, and the following line throws an error

NoMethodError: [!] undefined method `delete_element' for nil:NilClass

This is occurring with us for version 3.10.1 but the code is unchanged in master.

I don't know precisely why this issue is happening now; we did just upgrade our fastlane to the latest version though.

Complete output when running fastlane, including the stack trace and command used
Full stack trace 
NoMethodError: [!] undefined method `delete_element' for nil:NilClass
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/helper/html_test_report.rb:189:in `block in remove_duplicate_testcases'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/helper/html_test_report.rb:186:in `each'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/helper/html_test_report.rb:186:in `remove_duplicate_testcases'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/helper/html_test_report.rb:26:in `each'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/helper/html_test_report.rb:26:in `collate_report'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/actions/collate_html_reports.rb:14:in `block in run'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/actions/collate_html_reports.rb:13:in `each'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/actions/collate_html_reports.rb:13:in `run'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/helper/multi_scan_manager/report_collator.rb:89:in `collate_html_reports'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/helper/multi_scan_manager/report_collator.rb:29:in `collate'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:250:in `collate_batched_reports_for_testable'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:198:in `block in collate_batched_reports'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:197:in `each'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:197:in `collate_batched_reports'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:179:in `run_test_batches'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:70:in `run'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-plugin-test_center-3.10.1/lib/fastlane/plugin/test_center/actions/multi_scan.rb:34:in `run'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/runner.rb:261:in `block (2 levels) in execute_action'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/actions/actions_helper.rb:50:in `execute_action'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/runner.rb:253:in `block in execute_action'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/runner.rb:227:in `chdir'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/runner.rb:227:in `execute_action'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/runner.rb:157:in `trigger_action_by_name'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/fast_file.rb:159:in `method_missing'
  Fastfile:478:in `block (2 levels) in parsing_binding'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/lane.rb:33:in `call'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/runner.rb:49:in `block in execute'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/runner.rb:45:in `chdir'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/runner.rb:45:in `execute'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/lane_manager.rb:47:in `cruise_lane'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/command_line_handler.rb:36:in `handle'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/commands_generator.rb:108:in `block (2 levels) in run'
  /Users/jenkins/.bundle/jenkins/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in `call'
  /Users/jenkins/.bundle/jenkins/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in `run'
  /Users/jenkins/.bundle/jenkins/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in `run_active_command'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in `run!'
  /Users/jenkins/.bundle/jenkins/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in `run!'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/commands_generator.rb:352:in `run'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/commands_generator.rb:41:in `start'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/fastlane/lib/fastlane/cli_tools_distributor.rb:119:in `take_off'
  /Users/jenkins/.bundle/jenkins/gems/fastlane-2.150.0.rc4/bin/fastlane:23:in `<top (required)>'
  /Users/jenkins/.bundle/jenkins/bin/fastlane:23:in `load'
  /Users/jenkins/.bundle/jenkins/bin/fastlane:23:in `<top (required)>'
lilyball commented

Looks like this may actually be a "nearly 100% issue" for us rather than an intermittent issue, when running our full UI test suite.

lyndsey-ferguson commented

Hello, @lilyball. This is my top priority for multi_scan at the moment as it is a crash that is disrupting any use of the tool.

While I work on this, can you look at this Issue? Someone with a similar error was able to provide the logs and it seems that some outside process was actually interfering with the production of the HTML files -- they were malformed.

I don't have the branch that Zardazare used to test, but you could disable collation of html files and examine the files for validity by providing false for the collate_reports option in multi_scan.

lyndsey-ferguson commented

@lilyball, can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-250-html-collation-crash"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

lilyball commented

I ran a full build of the test suite with your modified plugin, no failure. I then ran a build with collate_reports: false so I could inspect the individual html files. There's a lot so I couldn't look at everything, but every test failure I looked at had associated failure details.

So this doesn't actually tell me what was failing. However, PR #251 should fix the actual crash, with the unless failure_details.nil? check.