citation-style-language/Sheldon

For failing tests, add messages based on specific test failures

adam3smith opened this issue · 5 comments

adam3smith commented

I'm not sure if this is possible, but it'd be really cool to have Sheldon take the Travis test failures and translate them into human-readable messages in the post.

So if you have a failed test message such as this:

😟 There are some issues with your submission. Please check the test report for details.

Turn that into:

😟 There are some issues with your submission. Please check and fix the following issues:

See above for how to edit your pull request.

You can see the full test report here.

rmzelle commented

Big +1.

Travis CI seems to be aware that build errors are hard to find in build reports (https://blog.travis-ci.com/2019-04-01-travis-community-survey-results-2018-3: "we asked what you’d like to see improved at Travis CI. ... From these, we’d like to share the following common themes: ... UI Improvements. Specifically, making it easier to find build errors, and improving navigation were often mentioned."

We also played around a little with https://www.travisbuddy.com/ (see also citation-style-language/styles#3722), which can post build errors directly into a GitHub thread, but that would still not be as nice and readable as @adam3smith mock-up above.

retorquere commented

How easy this is to do depends a lot on what you use to do your tests -- behave and other gherkin-ish test suites often can spit out JSON output in addition to formatted output, and that JSON is specifically meant to be consumed for further processing. From the looks of it, the travis test uses rspec, and I think that can output json.

retorquere commented

The Sheldon update already added this, right?

rmzelle commented

The Sheldon update already added this, right?

Right :).

rmzelle commented

@adam3smith, if you find that any errors use confusing language, we should probably just address that directly in the RSpec code.