support "ignored" attribute

[garfieldnate]

It's useful to be able to specify if a test case failed but was expected to do so. This is different from the test being disabled. JetBrains uses the status="ignored" attribute on TestCase elements to indicate this.

[garfieldnate]

Ideally the output statistics would be different as well. With using the "disabled" attribute on a test that fails, I get this report:

381 tests run, 381 passed, 0 skipped, 0 failed.

Ideally I would want to know how many tests are failing but being ignored anyway.

[garfieldnate]

Sorry, my previous comment was not accurate. Given XML for a test case that fails and is also marked "disabled", the failure is not ignored currently.

        <testcase classname="MiscTests - OS X" name="testSmemArithmetic" status="disabled">
                <failure type="Test Failure">Assert: Boolean true check failed.</failure>
        </testcase>

The output for my build was:

381 tests run, 379 passed, 1 skipped, 1 failed.

[mikepenz]

@garfieldnate thank you for the report.

Usually if tests are disabled -> they won't be run at all, as such no failure element will be within them. If you remove the failure block in it, it would only be counted as skipped.

The ignored flag is an interesting element, but probably requires more changes to handle it accordingly.

[garfieldnate]

Thank you for the changes! Is this summary correct?

• tests with "ignored" attribute are now counted as skips (not tested but the code change is there)
• tests that are both failed and skipped no longer count towards the failure count.
• there are no separate stats output for failed ignored or passed ignored tests.

[mikepenz]

Yes, your summary is correct.