mikepenz/action-junit-report

Wish: Add Test Result Summary

markstos opened this issue · 6 comments

markstos commented

Github Actions now supports publishing "summaries". An example given is summarizing a test suite run:

https://github.blog/changelog/2022-05-09-github-actions-enhance-your-actions-with-job-summaries/

It would be great if support was added for that. I think it would address #518.

mikepenz commented

I am not sure how quick I will find to dig into this new functionality. If somebody is faster to look into that, happy to review a PR :)

mikepenz commented

A first version will be available in an upcoming beta release: #566

Would love to get more feedback which information people would love to see for such a summary

markstos commented

Thanks for this @mikepenz ! I like the examples seen here: https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/ Which show a table with

  • Test Result (Pass or Fail)
  • Passed (Int)
  • Failed (Int)
  • Skipped (Int)
  • Time Duration

Hmm, I think Github Actions already shows us how long each step takes to run, so we don't needs to repeat that here. Just the number passed, failed, and skipped (if available) would be helpful.

mikepenz commented

@markstos right now the action combines all results into a unified report.

The example table you show seems to split per report though, so that would be something to investigate.
As such for now the summary would be only passed (count if enabled), failed, skipped:
https://github.com/mikepenz/action-junit-report/actions/runs/2358187444#summary-6524519796

markstos commented

@mikepenz It's simple, but I think the content you have now is sufficient. My only feedback is to present the result in columns instead of rows so it's more compact with a bit less scrolling. Keep the emojis!

Test Result Passed Failed Skipped
Fail 121 1 0
markstos commented

Although maybe including "pass/fail" is redundant as well, because Actions will already be reporting the stage as passed/failed already.

Mainly, I think it adds some value to surface the number of tests run to keep an eye on how the test suite size is changing over time or to get an estimate of the average time per test.

(Maybe the average time per test would an interesting metric to add!).