karolsluszniak/ex_check

Executive Summary

Closed this issue · 3 comments

asummers commented

My use case is that I would like to lift the end summary of which checks ran/passed/failed and post them into a comment back to the PR to alleviate the needs for developers to log into CI to see what failed. Because I run my CI checks centrally for many projects, I worry that one or more projects will be in a state where a skipped check will not be apparent to users from e.g. not having credo installed.

Would you be interested in something like a e.g. --summary-to-file opt that allowed me to redirect this to a file/env var or some mechanism by which I can export this data somewhere?

karolsluszniak commented

Hmm, so you'd basically like to run as usual but also produce the artifact / log with summary - i.e. info about passed and failed checks? Sounds reasonable, but there are a few questions to answer:

  1. What format do we produce the output in? Is there any standard format that we could go for e.g. JUnit or simply HTML / text? It'd be nice to avoid adding deps like a JSON lib since currently ex_check intentionally has none to stay lightweight & non-invasive.
  2. Do we also include output from specific checks? If so, in the same file or separate ones?
  3. Do we want to allow switching the format, e.g. to enable CI integration - some CIs allow JUnit files there.

Aside from that, there are a few things that I don't understand in your described use case:

  1. How do you intend to produce a PR comment out of the summary file? Maybe that should be a part of ex_check too?
  2. How would producing such summary file save your developers from skipping credo? Perhaps you could simply check for that e.g. by parsing mix.lock in projects?

Please elaborate :)

asummers commented

For JSON, I the general pattern is to do a check for e.g. Jason before spitting it out, but in the interest of making it extensible, maybe allowing a user to specify an output format module would be a good future goal if need arises, so someone could export as HTML, JSON, XML, whatever. For my purposes, a plain text comment is completely sufficient, and I don't need any external dependencies.

Having ex_check write that to a file would be useful. Then it's just snaking that output file into postprocessing as needed and eventual curl calls. For credo, I was more talking about seeing the visual feedback in the output on the PR of "skipped due to not having credo", but I realize I can simply take off the detect to make it run always. Nonetheless, having the users be able to stay on GitHub and see the failures without having to go to CI and see the summary would be useful to others besides myself, I think =)

karolsluszniak commented

Sorry for taking some time, but I'm coming back with a good news. I've added support for producing manifest file in #24. Since it's plaintext that includes list of all passed, failed and skipped checks, and since --manifest command line option allows to specify the file path, it should cover your needs from this issue.

Btw, my motivation for doing this was to allow to only run failed tools with --failed command line option, just like ExUnit allows. But it's great to kill two birds with one shot :)

I'll merge and release v0.13 with this addition as well as all others from you in next few days.

  • 🎉1