Include better failure information in swift-test xunit output

[GNMoseke]

Description

Problem

Currently, when creating an xunit xml output file with the --xunit-output option in swift test, test failures generate an empty <failure> tag:

 <testcase classname="ExampleTests.Tests" name="testFail" time="0.041534993">
 <failure message="failed"></failure>
 </testcase>

This doesn't really provide much useful information for anything that may want to consume this information (CI being the prime example). The xunit xml spec allows any string to be inside this tag.

I am not intimately familiar with XCTest, so as the fixme mentioned below indicates it may be that there just needs to be better reporting given by XCTIssue to include more relevant details.

Possible Fix

As the FIXME comment here suggests, we could simply stick the result.output onto line 1284

"""
            if !result.success {
                content += "<failure message=\"failed\">\(result.output)</failure>\n"
            }
"""

which generates XML that looks like this (just has stdout from the test invocation mushed into it):

<testcase classname="ExampleTests.Tests" name="testFail" time="0.029471092">
<failure message="failed">Test Suite 'Selected tests' started at 2024-06-01 21:02:59.244
Test Suite 'ExampleTests' started at 2024-06-01 21:02:59.246
Test Case 'Tests.testFail' started at 2024-06-01 21:02:59.246
/home/test-ex/Tests/test.swift:8: error: Tests.testFail : XCTAssertEqual failed: ("1") is not equal to ("2") -
/home/test-ex/Tests/test.swift:9: error: Tests.testFail : XCTAssertEqual failed: ("foo") is not equal to ("bar") -
/home/test-ex/Tests/test.swift:10: error: Tests.testFail : XCTAssertTrue failed -
Test Case 'Tests.testFail' failed (0.001 seconds)
Test Suite 'ExampleTests' failed at 2024-06-01 21:02:59.247
     Executed 1 test, with 3 failures (0 unexpected) in 0.001 (0.001) seconds
Test Suite 'Selected tests' failed at 2024-06-01 21:02:59.247
     Executed 1 test, with 3 failures (0 unexpected) in 0.001 (0.001) seconds
<failure>

That isn't ideal, but it's at least more detail than is given currently

If the full output is too verbose, there's plenty of room to process it down to something cleaner, but that would obviously be beholden to the output format remaining the same.

This does seem generally like it may need a wider refactor to pull additional information from the test reporter though, adjacent issues:

• #4752
• #7383

Expected behavior

xunit.xml failures include relevant failure information

Actual behavior

xunit.xml failures are empty

Steps to reproduce

Run swift test --parallel --xunit-output test.xml in any project with a test failure

Swift Package Manager version/commit hash

5.10

Swift & OS version (output of swift --version && uname -a)

Swift version 5.10 (swift-5.10-RELEASE)
Ubuntu 20.04 & macOS

[grynspan]

Note that the output schema more closely matches JUnit.