Annotation truncates failure output/callstack

[stackptr]

It appears only the first few lines of a failure output are used when this action creates an annotation.

The action is configured in a workflow following a step that produces a JUnit report:

     - uses: mikepenz/action-junit-report@v4
         if: success() || failure()
         with:
           report_paths: '/tmp/junit/**/test_results.xml'
           annotate_only: true
           test_files_prefix: backend/

An XML file in the report_paths looks like¹:

<testsuites name="fancy-api">
  <testsuite name="Freckle.Api.Handlers.Ela.AdaptiveSkill.Passage//ela/adaptive-skills/assignments/:id/passages GET" package="Freckle.Api.Handlers.Ela.AdaptiveSkill.Passage//ela/adaptive-skills/assignments/:id/passages GET" id="0" time="1.020266000" timestamp="2024-01-08T22:27:40.014365Z" hostname="localhost" tests="3" failures="1" errors="0" skipped="0">
    <properties/>
    <testcase file="fancy-api/tests/Freckle/Api/Handlers/Ela/AdaptiveSkill/PassageSpec.hs" line="193" name="does not return passages that are not related to the student's current skill" classname="Freckle.Api.Handlers.Ela.AdaptiveSkill.Passage//ela/adaptive-skills/assignments/:id/passages GET" time="0.335951000">
      <failure type="error">fancy-api/tests/Freckle/Api/Handlers/Ela/AdaptiveSkill/PassageSpec.hs:193:11
Graphula with seed: 3059625118439811851

Graph dumped in temp file: /tmp/nix-shell.PMT5JB/graphula/fail-65038-0.graphula

Expected status was 400 but received status was 200. For debugging, the body was: {"id":1,"assignmentId":1,"subject":"ela","teacherId":3,"studentId":3,"status":"completed","createdAt":"1858-11-15T00:00:00Z","title":"","completedAt":"1858-11-17T00:00:03Z","accuracy":1.4229479,"numQuestionsAnswered":50,"isSelfAssigned":"teacher_assigned","metadata":{"tag":"ela_adaptive_skills_practice","contents":{"domain":{"id":"0a50db28a6d2d54e24d02ebe59149721","name":"\u0013s';\u0003","lowestGrade":5,"highestGrade":10,"k5ThumbnailImage":"|","k5PathwayImage":"\tk","msThumbnailImage":"\u0019􁫹","msPathwayImage":"","hasElaPathwayContent":true,"hasElaGrammarQuestions":true,"hasElaWordStudyQuestions":true,"hasElaArticleWritingQuestions":true,"hasElaArticleReadingQuestions":false,"isStudentVisible":true},"rlStandard":{"id":"b6dde810a09f51ff1a50f18f1ff34b2c","name":"","shortName":"\u001e0kZ","description":"J󿾯\u000e=]ࡂ#","progressionOrder":0,"grade":5,"domainId":"0a50db28a6d2d54e24d02ebe59149721","hasElaPathwayContent":false,"hasElaGrammarQuestions":false,"hasElaWordStudyQuestions":true,"hasElaArticleWritingQuesti... (use `printBody` to see complete response body)</failure>
    </testcase>
</testsuites>

However, the annotation only contains the first few lines:

Using gh api to get the annotations for the commit shows the same truncation, i.e., that there does not seem to be an issue rendering the annotation in the GitHub UI:

[
  {
    "path": "backend/fancy-api/tests/Freckle/Api/Handlers/Ela/AdaptiveSkill/PassageSpec.hs",
    "blob_href": "https://github.com/freckle/megarepo/blob/d64c66d1b28bdfd2463248fd2edebdcd9f95376e/backend/fancy-api/tests/Freckle/Api/Handlers/Ela/AdaptiveSkill/PassageSpec.hs",
    "start_line": 193,
    "start_column": null,
    "end_line": 193,
    "end_column": null,
    "annotation_level": "failure",
    "title": "fancy-api/tests/Freckle/Api/Handlers/Ela/AdaptiveSkill/PassageSpec.hs.does not return passages that are not related to the student's current skill",
    "message": "fancy-api/tests/Freckle/Api/Handlers/Ela/AdaptiveSkill/PassageSpec.hs:193:11\nGraphula with seed: 8221632655132446637",
    "raw_details": ""
  }
]

GitHub API documentation for creating a check run states that there is a 64KB limit for output.annotations[].message or output.annotations[].raw_details. Can this action use the entirety of the test failure output when creating the annotation?

Footnotes

1. This file is from the artifact created from /tmp/junit by the workflow, with irrelevant test cases removed. ↩

[stackptr]

After some digging, I think this is the source of my issue:

const message: string = (
  (failure && failure._attributes && failure._attributes.message) ||
  (testcase.error && testcase.error._attributes && testcase.error._attributes.message) ||
  stackTrace.split('\n').slice(0, 2).join('\n') || // <- here
  testcase._attributes.name
).trim()

It looks like message is constructed by taking the the first few lines from stacktrace. stacktrace appears to be sent as raw_details in the annotation, but this is an empty string when using gh api to see the annotation. Given that the contents of the <failure> will include randomly generated text that may be various bit of Unicode, the escapeEmoji function could be the culprit, but I don't see any false positive when testing the sample report (above) against the regex replacement.

[stackptr]

Repurposing an existing test using the sample JUnit report, it seems like the testParser is behaving, producing an annotation with raw_details matching the contents of the <failure>.

[stackptr]

I see that while Annotation contains raw_details: string, it is dropped when create core.AnnotationProperties. It doesn't seem like core.error provides any way to set this. I will try using the alternate API via the update_check input and see if that is any different.

[mikepenz]

@stackptr thank you for the detailed report.

Your observations are correct, the message used in the reports is prioritising the message property in the report, and only if that is missing falling back to the stacktrace - shortened to 2 lines (the body of the failure, if that's not there body of the testcase)
https://github.com/mikepenz/action-junit-report/blob/main/src/testParser.ts#L321-L326

The action currently while keeping track of the raw_details (which is the raw stacktrace) doesn't use it to report it.

It would require change sin the https://github.com/mikepenz/action-junit-report/blob/main/src/annotator.ts to report the raw_details in addition / instead)

[stackptr]

After some experimentation, I found annotations added via annotate_only: true to be more reliable than those created by adding or updating check runs. I opened #1007 to allow for annotations that include the full failure output, but preserved the default of truncating that to two lines, since it seems likely that this output is usually a stack trace, and full output is not needed.

[mikepenz]

Released v4.0.4 with your PR. Thank you very much for your contribution!