Write test results to a file

[adolgert]

Would it be possible to write test results to a file? I would like to use the passed and failing tests to do test selection. The file should show each test case separately and each call of the test case with different parameters separately.

Maybe a specific file format would work well: There is a standard file format for unit test reporting, called JUnit XML. It's the file type that the TestReports.jl package makes for results from Base.Test. It would be very nice to be able to run Jute.jl from the same Github Actions that use TestReports.jl, but that would require refactoring TestReports.jl to work with Jute.jl, as well as Base.Test. This seems like too much to ask right now.

I'm thinking about implementation, if I may help. TestReports.jl has an example of writing the XML file in the correct format. We could mimic that in Jute.jl, if that works for you. I'm not sure, but I think the place to hook to get this information would be in runtests.jl:run_testcases. Each testcase returns an outcome which has a result, a time, and a string. This could be sent to an observer that writes XML.

Thank you!

[adolgert]

I was mistaken. It's easier. ReportTests.jl expects to run a runtests.jl that has @testset declarations. Jute.jl does have @testset declarations. I made the JuteTestSet obey the usual rules in the record function, and now it runs under ReportTests.jl, which means it writes an XML file. There's more to do for a good PR, but this is promising.

[fjarri]

That's great, I almost forgot that I reused the Base.Test machinery. That will indeed make things easier. Does JuteTestSet have to be inherited from ReportingTestSet for the reports to work?

[adolgert]

It doesn't. The requirements, from the ReportTests.jl side, are that the testset have a description and results:

mutable struct JuteTestSet <: Test.AbstractTestSet
    description :: AbstractString
    results :: Vector

    JuteTestSet(descr; results=[]) = new(descr, results)
end

This means the test case has to be invoked with a name argument:

        Test.@testset JuteTestSet "$(tc.name)" results=:($results) begin

It also has to report its tests to any parent test cases when it's finished. So that has to be added.

function Test.finish(ts::JuteTestSet)
    # If this runs under ReportTests, tell the parent testset what happened.
    if get_testset_depth() != 0
        record(get_testset(), ts)
    end
    ts
end

From there, it's all clear.