Test count inconsist between native julia and xml report

[hofmannmartin]

Thank you for this package. It has been a real help.

I may have stumbled upon one inconsistency. To this end let us consider the following @testset

using Test

@testset "Example" begin
    @test variableThatDoNotExits == 42     
end

which reports 1 test and 1 error

Test Summary: | Error  Total
Example       |     1      1

---

If I generate a test report for said test set

using Test, TestReports

(@testset ReportingTestSet "Example" begin
    @test variableThatIsNotDefined == 42     
end) |> report |> println

I get

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="0" failures="0" errors="1">
<testsuite name="Top level tests" tests="0" failures="0" errors="1" time="0.140" timestamp="2021-06-18T14:25:29.083" hostname="moeddel-ThinkPad-T460s" id="0">
<error message="UndefVarError: variableThatDoNotExits not defined" type="UndefVarError" classname="Top level tests" time="0.140">
UndefVarError: variableThatDoNotExits not defined
Stacktrace:
 [1] macro expansion
   @ REPL[9]:2 [inlined]
 [2] macro expansion
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
 [3] top-level scope
   @ REPL[9]:2
</error>
</testsuite>
</testsuites>

which reports 0! tests and 1 error.

I would expect 1 test and 1 error too, or am I on the wrong track?

[mmiller-max]

Thanks for raising this, it's a good spot! I think we are currently not distinguishing between errors that happen in a test and errors that happen out of a test, e.g.:

@testset "Example" begin
    @testset "1" begin
        variableThatDoNotExits
    end
    @testset "2" begin
        @test variableThatDoNotExits == 42     
    end
end

Which gives

Test Summary: | Error  Total
Example       |     2      2
  1           |     1      1
  2           |     1      1

Interestingly, the standard output does not differentiate between the two either.

The two errors in the above example can be distinguished however. The Error object for the first one has :nontest_error in it's test_type field, where as the second has :test_error. We could count it as a test when it's the latter, but not when it's the former. So I think we can improve the behaviour to count a test when an error occurs during the test, but even so it won't always match the Test Summary output as it seems to count non test errors too.