Providing unit test results in Sphinx documentation

[Paebbels]

Hello @LarsAsplund.

I worked on a new Sphinx extension, which can render junit XML files as tables in a Sphinx based documentation.

This would allow you to:

• display Python code test results of VUnit (I think you run pytest - so either you have that file already, or it's easy to generate)
• display VHDL testbench results as VUnit can produce junit XML files.

Could you provide me a junit output of VUnit, so I can test my extension?

Moreover, you can find 2 other report formats in my extension that could be interesting for you:

1. display code coverage from cobertura files. This can be either of VUnits Python sources from unit tests, but also test results e.g. from an offline VUnit + Riviera-PRO run (data from Riviera-PRO can be converted to Cobertura).
2. display documentation coverage of VUnits Python code.

See for https://pytooling.github.io/sphinx-reports/index.html more details

[Paebbels]

This is my current state for rendering junit XML files via Sphinx:

[LarsAsplund]

@Paebbels VUnit supports two JUnit XML dialects: Jenkins and Bamboo. I attached the results for our basic user guide example that has both passing and failing test cases.

user_guide.zip

[Paebbels]

@LarsAsplund Thanks for these files.

Are you aware that these files (formats) are not JUnit compliant? Do you have a format specification (XSD, DTD, ...) for the Jenkins and Bamboo dialects?

For "original junit" I researched this: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd
As a JUnit overview I also found this: https://github.com/testmoapp/junitxml

Problems are:

• as this is an XML, a XML processing instruction should be given:
  <?xml version="1.0" encoding="UTF-8" ?>
• The document root tag <testsuites> is missing.
• The timestamp is missing in the root element.
• The execution time is not accumulated in the testsuite nodes.
• Whitespace is not preserved (<... xml_space="preserve">...</...>)

As VUnit knows the number of assertions, it could be exported too.

[LarsAsplund]

I did not. When we first released VUnit almost a decade ago there was no definition to be found. We simply had to look what worked reasonably well in practice. The way Jenkins was doing it seemed to work fairly well for most other tools but at some point we were made aware of Bamboo doing it differently and we introduced the option to select what dialect to use. Haven't look into it since then. I will look at that specification you provided. Are you saying that this format is widely acknowledged or is it just an effort to reconsile the differences between dialects?

[Paebbels]

According to some sources, the XSD by Windyroads refers to the original syntax of JUnit as produced by Java Ant. I also know that there are multiple JUnit versions and JUnit has writer classes to produce the XML. Unfortunately, the official documentation doesn't contain any syntax description for new and old versions what such a class would write. I'm until now refusing to read the Java code to reverse enginier what the could write to extract the syntax from it. So I hope that XSD is correct.

OTOH, many have butchered the format and introduced simplifications like:

• Remove <testsuites> if there is only a single <testsuite> element.
• Add more attributes if you want to store more data like <skipped type="mark.skipped"> in pytest.

Neither of such modifications are allowed by XML / XSD. XML defines a tree structure and each node has certain attributes that must exist or are optional. This is needed so an XML file can be verified by the parser and a user can rely on a certain structure ... so less if element exist questions in the code.

So I strongly refuse the common practice by Jenkins to allow the removal of <testsuites> and moving attributes like hostname to the testsuite level. XML is either way a talkative format, why spare 50..100 bytes on a several kB output format.

Then I found the testmoapp repository any they claim to have collected/recreated what the market produces/accepts as formats. The also found these additions with <properties>. What ever tool does this, that would be a nice way of adding additional data without breaking the basic XML structure. So the XML syntax can focus on suites and cases as well as results and outputs, but doesn't care about user defined annotations.

I'm not yet sure (need to refresh my XML knowledge) if an XSD can already define a tag as whitespace preserving, if not, then the xml:space="preserve" attribute are required, otherwise whitespace will be stripped by parsers and all commandline output are broken. (Or Jenkins, Bamboo, etc. implement handwriten non XML standard comformant XML parsers ... wohu ... I hope not.)

My suggestion to VUnit would be, to add the outer <testsuites> document node and move the hostname attribute, to avoid that dirty Jenkins/Bamboo hack. Also please add the XML processing instruction <?xml ...?>.