JUnit files are appended to with RSpec formatters

Hello, found an interesting case while using this gem.

I'm trying to add JUnit formatter to use the results in Jenkins using the junit instruction. When using queue mode with this gem (and service) however, Knapsack might run rspec tests multiple times from the same process. This is causes a problem because --out in RSpec appends to a file. So that means you end up with a file that has something along the lines of:

<xml ...>
<testcase stuff in here />
<xml> # next suite run starts and appends...

Jenkins (and from what I can see, XML parsers in general) don't like when there are multiple <xml> tags in the file, as you can see from this error:

Failed to read test report file /workspace/HCM_PR-7675-355EZNVL3NLOS25MQ7X2QIWUKL7VGXOWVAWXENJHPGTJ4IJEY4FA@5/tmp/rspec_results7c4385c84.xml
org.dom4j.DocumentException: Error on line 2175 of document file:///workspace/HCM_PR-7675-355EZNVL3NLOS25MQ7X2QIWUKL7VGXOWVAWXENJHPGTJ4IJEY4FA@5/tmp/rspec_results7c4385c84.xml : The processing instruction target matching "[xX][mM][lL]" is not allowed.

The more important piece is this:

The processing instruction target matching "[xX][mM][lL]" is not allowed.

Detail on the error is in this stackoverflow: http://stackoverflow.com/questions/19889132/error-the-processing-instruction-target-matching-xxmmll-is-not-allowed

I can see an "after hook" of sorts be helpful here because we could just rename the file afterwards to something that won't be appended to and Jenkins can read the multiple files just fine.

Thoughts?

Hey @bobbytables

I could create custom JUnit formatter. I did similar thing for summary:
https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/lib/knapsack_pro/formatters/rspec_queue_summary_formatter.rb
in this PR:
https://github.com/KnapsackPro/knapsack_pro-ruby/pull/31/files

After that the formatter could be specified in the knapsack command similar as here:
https://github.com/KnapsackPro/knapsack_pro-ruby#how-to-use-junit-formatter

I've tried the approach with hook and I did an example with hook after a subset of tests executed from the queue to rename the output rspec.xml file but then I figure out that actually the junit formatter cumulate results so each new rspec.xml has inside of it all results from previous subset queue run and there are multiple opening xml tags.

Related PR: #41

So it seems the only way would be to write custom junit formatter then. I will look into that or I will fork the gem to adjust it to knapsack_pro case.

Well with the ability to have an "after" hook, you could just rename the file. That way the next run just creates a new file.

Yes, that's what I did but the problem is with the way how formatter works when I execute multiple times the RSpec::Core::Runner for subset of work queue.

knapsack_pro-ruby/lib/knapsack_pro/runners/queue/rspec_runner.rb

Line 32 in fc911fa

exit_code = RSpec::Core::Runner.new(options).run($stderr, $stdout)

The junit formatter keeps in memory xml from previous run so the next rspec.xml file has double xml tags even when the first rspec.xml file was renamed.

I did custom formatter for RSpec Queue Summary so I will have to do a similar thing for junit formatter.
#31

Is there a way to force the formatter to write out?

This is how junit formatter dump xml to file:
https://github.com/sj26/rspec_junit_formatter/blob/master/lib/rspec_junit_formatter/rspec3.rb#L19
https://github.com/sj26/rspec_junit_formatter/blob/master/lib/rspec_junit_formatter.rb#L18

I noticed when I use just junit formtter without my "after" hook then sometimes the xml file has weird content like the content was incorrectly appended to the file.

When I did rename rspec.xml file after each subset of work queue then the last generated rspec.xml file has correct content with duplicated xml tags. The last one xml tag contains the full report. So it means we would have to remove from rspec.xml the content till the last opening xml tag.

Here is short example of how last generated rspec.xml file looks like.

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="rspec" tests="1" failures="0" errors="0" time="0.027553" timestamp="2017-05-23T23:02:03+02:00">
  <!-- Randomized with seed 24503 -->
  <properties/>
  <testcase classname="spec.features.calculator_spec" name="Calculator when try to add without provided numbers result is 0" file="./spec/features/calculator_spec.rb" time="0.094628"/>
</testsuite>
<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="rspec" tests="2" failures="0" errors="0" time="0.027553" timestamp="2017-05-23T23:02:03+02:00">
  <!-- Randomized with seed 24503 -->
  <properties/>
  <testcase classname="spec.features.homepage_spec" name="Homepage Features has welcome text" file="./spec/features/homepage_spec.rb" time="0.197960"/>
  <testcase classname="spec.features.calculator_spec" name="Calculator when try to add without provided numbers result is 0" file="./spec/features/calculator_spec.rb" time="0.094628"/>
</testsuite>

I will play more with this idea tomorrow.

Sounds good, thanks for looking at this so intently 😄

@bobbytables I figured out this. Could you update knapsack_pro version and add the hook as described here:
https://github.com/KnapsackPro/knapsack_pro-ruby#how-to-use-junit-formatter-with-knapsack_pro-queue-mode

Basically, we need to rename the rspec.xml in after subset queue so the file won't be appended with duplicated xml content. The last execution of tests subset from work queue will create the rspec.xml with xml that contains the tests from all subset runs (all tests from the particular CI node).

Please let me know if this works for you.

The link in the previous comment no longer works. Looks like that information now lives here: https://knapsackpro.com/faq/question/how-to-use-junit-formatter#how-to-use-junit-formatter-with-knapsack_pro-queue-mode

Here is the current docs page related to formatters: https://docs.knapsackpro.com/ruby/rspec/#formatters-rspec_junit_formatter-json