[question] Where to report the filename and line number in JUnit?
carlescufi opened this issue · 14 comments
Note: This is not a bug report, just a usage question.
The JUnit spec doesn't mention file name and line number where an error occurred (eg. as attributes of failure).
Since this action seems to display annotations inline with the PR files changed (i.e. the diff), how can one provide the file name and line number where an error was detected (for example by a linter program).
@carlescufi if available the action will retrieve the line and file from the xml file directly via it's attributes: https://github.com/mikepenz/action-junit-report/blob/main/src/testParser.ts#L294-L299
It will try to do some searching based on the retrieved file to match it with the location of the source.
If it is not directly defined, it will try to find the line number by parsing it from the fileName (often the line number is defined after the file-name: https://github.com/mikepenz/action-junit-report/blob/main/src/testParser.ts#L53-L92
This xunit report for example has it defined as part of the report:
@mikepenz thanks very much for the reply.
@carlescufi if available the action will retrieve the line and file from the xml file directly via it's attributes: https://github.com/mikepenz/action-junit-report/blob/main/src/testParser.ts#L294-L299
The issue is that, unlike I'm misunderstanding it, this is a testcase attribute, and not a failure one. In my case there can be multiple failure nodes for each testcase, and each will have its own file name and line number. This is because the testcase is "lint" and then the linter can find multiple issues in the code.
EDIT:
In fact the spec does include filename and line number per failure, somewhat:
<failure message="PROGRAM.cbl:2 Use a program name that matches the source file name" type="WARNING">
WARNING: Use a program name that matches the source file name
Category: COBOL Code Review – Naming Conventions
File: /project/PROGRAM.cbl
Line: 2
</failure>
Your info potentially raises a very different question.
You note there are multiple failures per testcase? Currently the action only creates one annotation per testcase, which means that it would not properly handle your report?
You note there are multiple
failuresper testcase? Currently the action only creates one annotation per testcase, which means that it would not properly handle your report?
That's correct @mikepenz, that's how I use the JUnit format. For now I am annotating manually but I wanted to switch to an action like yours, but of course that's not gonna work if it doesn't support multiple failure XML elements per testcase. I am really not sure if this is allowed or not, apparently the spec is not quite clear?
Up until now there was no use-case to support multiple failure blocks within a single testcase and I am not certain if this is allowed by the spec.
To be honest, the action itself does support more formats than just plain junit as you can see with the different test files we cover as part of the tests: https://github.com/mikepenz/action-junit-report/tree/main/test_results
There are surely enough interpretations.
Not sure if there is a case to expand the handling for your described use-case or if it would be better to go with a fork 🤔
Is there a format close to junit which allows for multiple failure nodes?
Up until now there was no use-case to support multiple failure blocks within a single testcase and I am not certain if this is allowed by the spec.
Agreed, I am not sure either as mentioned.
To be honest, the action itself does support more formats than just plain junit as you can see with the different test files we cover as part of the tests: https://github.com/mikepenz/action-junit-report/tree/main/test_results
Would you be so kind as to point me to a format that actually supports multiple failures per testcase? Or do people create artificial test cases when a single tool run can fail multiple times in multiple places? (again think running a linter on the PR diff).
Thanks again!
Is there a format close to
junitwhich allows for multiplefailurenodes?
That's exactly what I was wondering... no idea!
I believe the junit format was not built to handle this situation, given that a test normally fails with a single error (but I may be wrong here :D)
For lint I used https://danger.systems/ruby/ in the past. E.g. use it to parse the lint result file, and then have it create the annotations.
There may be other actions though which directly support lint output.
Thanks for all the input @mikepenz, very much appreciated. Closing this question now.
@mikepenz I pulled the thread a bit more and found this:
which then links to:
or
Which seems to allow multiple failures in a single testcase.
🤔 wonder how that would ever be possible for a test
🤔 wonder how that would ever be possible for a test
Here's an example of the XML I generate:
<?xml version="1.0" encoding="utf-8"?>
<testsuites tests="1" failures="5" errors="0" time="0.0">
<testsuite name="Compliance" tests="1" errors="0" failures="5" skipped="0" time="0">
<testcase name="Checkpatch" classname="Guidelines">
<failure message="subsys/bluetooth/host/hci_core.c:269 do not use C99 // comments" type="error">
C99_COMMENTS: do not use C99 // comments
File:subsys/bluetooth/host/hci_core.c
Line:269</failure>
<failure message="subsys/bluetooth/host/hci_core.c:270 that open brace { should be on the previous line" type="error">
OPEN_BRACE: that open brace { should be on the previous line
File:subsys/bluetooth/host/hci_core.c
Line:270</failure>
<failure message="subsys/bluetooth/host/hci_core.c:275 space prohibited between function name and open parenthesis '('" type="warning">
SPACING: space prohibited between function name and open parenthesis '('
File:subsys/bluetooth/host/hci_core.c
Line:275</failure>
</testcase>
</testsuite>
</testsuites>