Print list of failed tests at the end of test run
Closed this issue · 50 comments
Do you want to request a feature or report a bug?
Request a feature
What is the current behavior?
Jest spits out each error individually without any summary at the end of run about which tests failed (individual files)
What is the expected behavior?
Just like now + a helpful summary of which tests failed.
Example how RSpec does this (see the "Failed examples" part)
Jest does this once you have enough test files for this to actually be useful, like 20 test suites or so.
Got 35 suites, made 2 suits fail, no summary.
@cpojer I got 5 suites. 12 tests total. Yet "pages" of logs. Being able to force a summary would be helpful.
Would you accept a PR that adds an option to force the summary ?
The output when run in cmd is longer than the buffer will keep for 6 failed tests. My experience is that this was something I really wished was implemented as soon as I wasn't able to see all the failed tests (so more than one) on the screen at the same time.
Threshold is set to 20: https://github.com/facebook/jest/blob/fb10f7a95161f3d93ec54be3b3f69359913b5691/packages/jest-cli/src/reporters/summary_reporter.js#L22
It seems to work fine for me, can somebody come up with a quick repro which shows the summary not being displayed?
An option for forcing the summary might make sense, but I'd like to see if we can't make it smarter first 🙂
The summary that it prints is not the summary that we're all looking for.
The current "summary" is all of the output of all of the failed tests. Each failed test can have arbitrarily long output, which makes it very difficult to actually figure out the distinct files that failed.
What we'd like in a summary is a single line per file (or perhaps per failing test), so that it's very easy to see exactly what failed without scrolling through hundreds or thousands of lines of text.
That sounds like a custom reporter to me
I'd also like to voice my support for a summary of failing tests/suites in the output. I have 68 failing tests, over 22 suites. The current output is too long for my terminal buffer, so it's very difficult to work through these failures (or even know which suites have failed!).
A short summary (simply a list of failing suites) at the end of the output would remedy this. I could then filter to a single suite at a time, fix, and move on. This workflow is unnecessarily difficult right now
Thanks for the bump @WickyNilliams, I think we can add a flag like compact which will show the suites that failed but not the failure message for each test
That would be amazing!
@SimenB An option for forcing the summary might make sense, but I'd like to see if we can't make it smarter first 🙂
Is there any issue to improve that? Or is there any other way to show all suites and tests?
This is the only reasonable google result for jest do not repeat failed test details at the end, so I'm asking for help here:
I would like to see failure message 1 time for each test, not 2 times per failure - it's OK to list failed test in the summary at the end, but I would like to turn OFF the duplicated multi-page details of a snapshot mismatch (of a React component with a lot of data in props)...
Is there an existing CLI option please? Or another issue to add the feature?
(I am using the Jest version included in CRA, "react-scripts": "1.1.4", but we can update if the option is available in 1.1.5)
Hey, so I ran in to the same issue and could not find a solution through google, so I ended up writing my own reporter jest-summarizing-reporter.
It can either print only the summary or print the test diffs at the top and then a summary.
Only when time came to publish to npm did I find out that at least 3 packages with same functionality already exist npm search results for jest summary reporters
If anyone finds this thread in same situation as I did, I hope this helps you.
P.S. There are at least 4 public implementations of this behaviour now and probably many more private ones.
Two top search results for jest summary reporter are this thread and one more where OP wants the same thing. Judging by these threads there is demand from the users.
Also if you look at the screenshot of the output of my reporter:
You can see that it's almost identical to the default one. The only real difference being that the failureMessage's are being printed before the full summary, not during it. So there is a lot of default reporter code duplication in the custom reporters - the colors, etc.
I honestly think this should be either a default behaviour or at least behind some option in jest.
I would love to try contributing if possible.
Happy to take a PR making it an option passed to the default reporter (note that it should be reporters: [['default', {config: 'here'}]]) and not part of globalConfig
Docs for reporter config: https://jestjs.io/docs/en/configuration#reporters-array-modulename-modulename-options
Hi @SimenB. Can I work on this issue?
For sure, go for it! 🙂
@SimenB How about adding this feature to summary reporter so that it will be shown at the end of the test results?
@doniyor2109, @SimenB If I understand correctly that SummaryReporter is intended for the summary that is only shown when number of test suites is > 20 and there is at least one failing test, then that is not the place for this functionality.
The root problem is that when there are a lot of failed tests, their diffs fill the terminal and make it almost impossible to understand what were the tests that failed. This can happen without there being more than 20 suites.
What is actually needed is a list of names of failed tests, in a convenient and easy to access place (like the end of the output). That's it.
Removing the 20 suite threshold and the diffs (leaving only test names) from the SummaryReporter might be a good option. That way there would also be no diff duplication (which is present now when the SummaryReporter kicks in)
@doniyor2109 Please make sure you read all of the comments in this thread and in this one as well #5358 and this denied PR #5676 before starting. Or let me implement this, I mentioned in my comment above that I would not mind doing it, but you kinda beat me to it with your first comment.
As a user, I'd prefer keeping current summary at the end. It's a discrete overview of information that can be consistently expected at the end of the run, if we put anything after it that varies in length, then you will need to search to find the summary which is a bad experience IMO
It would probably be good to submit an overview of proposed changes before making any work to avoid churn
Threshold is set to 20:
It seems to work fine for me, can somebody come up with a quick repro which shows the summary not being displayed?
An option for forcing the summary might make sense, but I'd like to see if we can't make it smarter first 🙂
Having ability to force the summary definitely makes sense.
For example, I have a use case, when I need to include an excerpt of a console output from CI server and send it via email every time the build fails. That way I am able to get a quick feedback about what went wrong just by looking at the email.
But since the number of test suits in my project is lower than 20 and the size of the log excerpt included in the email is limited to a reasonably small amount (100 lines), I only get the log of passed tests, while failed tests are truncated.
With existing custom summary-reporter libraries I'm not able to do this, because they just print which tests failed, but doesn't print "expected vs actual" message.
I know it is a kind of specific case, but I think it might be useful in more general sense: when you need to get summary of failed test cases and the reason why they failed even if you have less than 20 test suites
If you are frustrated that a readable report listing failing suites is not possible out of the box with jest-cli, this bash workaround for Mac OS and probably linux, throws away all output except the suite failure lines...
npx jest 2>&1 | grep 'FAIL'Any ETA on this?
I wonder how people even use jest in real world without having summary in the end :(
I found a little way to get the exact tests which failed:
Failed tests have this ● character at the beginning
Using @cefn's solution we can extract only the names of the failed tests!
npx jest 2>&1 | grep '●'
Works pretty good
@benwinding no, this doesn't works well, because without splitting and combining output you get whole output xor overview, but not both.
The solution from @cefn worked great for me.
I added this to our package.json: "test-compact": "jest 2>&1 | grep 'FAIL'", and now can just run yarn test-compact on the command line to see a list of the failed tests. Failed tests are shown twice, but nbd.
If you are frustrated that a readable report listing failing suites is not possible out of the box with jest-cli, this
bashworkaround for Mac OS and probably linux, throws away all output except the suite failure lines...npx jest 2>&1 | grep 'FAIL'
Thanks for answer. Addition to that, you can use
npx jest 2>&1 | grep 'FAIL' | sort -t: -u -k1,1for removing duplicate outputs.
I think most people just use a Jest VScode extension, which shows failed tests inline and does more helpful stuff, but no one mentioned that here.
I think most people just use a Jest VScode extension, which shows failed tests inline and does more helpful stuff, but no one mentioned that here.
That's super useful, thanks Nnadozie!
Nobody likes to be reminded of their failures. :-)
Since my VSCode terminal window has all the jest output, I'd rather not have the details of each failure repeated.
If there is going to be an option to control the failure summary, please could it include the option to disable it?
In the meantime, I'll just have to: sed '/^Summary of all failing tests/,/^Test Suites:/d'.
If you are frustrated that a readable report listing failing suites is not possible out of the box with jest-cli, this
bashworkaround for Mac OS and probably linux, throws away all output except the suite failure lines...npx jest 2>&1 | grep 'FAIL'
unfortunately this doesn't work on non-posix environment, like windows, that lack a "grep" implementation
npx jest 2>&1 | grep 'FAIL'
unfortunately this doesn't work on non-posix environment, like windows, that lack a "grep" implementation
These days I couldn't survive on Windows without the Unix distribution that comes with git (https://git-scm.com/downloads). That's what allows me to do the sed command I posted above (and a million other things).
If your solution has to work on vanilla Windows, there are equivalents - Windows has a find and findstr - but they're all horrible in one way or another compared to the Unix commands.
I was wondering if this feature was implemented.
some suggestions, if not resolved
- being able to set TEST_SUMMARY-THRESHOLD form cli as an argument
- being able to force showing summary
Might be able to solve this issue and make writing tests less boring :P
That Jest, by default does not print a summary of test failures in a report at the end is completely mind-boggling.
@sunfit or @doniyor2109 did either of you end up taking on implementing this?
I got this output at the end of running a single test file
However, all the output above that is a huge log of a JS exception, long enough to fill in my entire terminal scroll buffer. Hence, I have no idea which of the 25 test cases in that file are failing.
As @metawrap-dev said above, this is mind-boggling.
Seems to me the easiest fix is to add a CLI option that can override the hardcoded TEST_SUMMARY_THRESHOLD, i.e.:
jest --summaryThreshold=1
how does the above pull request actually solve this?
5 years gone and still Jest not doing what Mocha has been able to do for the past 10 years. And this is to show the most important information needed after failing tests. Why people even use Jest? It's slow, prints output when all test run not after each failure, cannot do a proper summary, mocking cumbersome, ...
tfw when you want to use an obvious feature and find out there's been an issue open about it for 5 years
I have 12 suites and one fails and I have to scroll up MANY-MANY screens to find out which test suite and test failed, this is very annoying and unproductive. Can we please get the summary of failed tests at the end no matter how many suites we have?!
I have 12 suites and one fails and I have to scroll up MANY-MANY screens
Hm.. 12 suites normally should print 12 lines, plus errors, plus summary. How did you manage to have many-many screens printed?
By the way, if 12 lines is still too much, you can use a minimal reporter like jest-silent-reporter.
#3322 (comment) life saver
It might be lame, but I run jest --watch and then choose an option to only rerun failed tests :)
Thanks!
Now I have a question. I'm sure the answer can be derived from the discussion, but would you be so kind to summarize here how to take advantage of the new feature? How are we supposed to set that new env variable to make the tests summary be printed at the bottom?
What "new env variable" are you talking about?
The issue got closed, because the 'summary' reporter now takes an option to configure the threshold of the summary of all failed tests. Before the threshold was hardcoded (with a value of 20), now you can configure it:
/** @type {import('jest').Config} */
const config = {
// the summary of all failed tests will be printed if there are ten or more failing tests
reporters: [['summary', {summaryThreshold: 10}]],
// or like this:
// reporters: [['default', {summaryThreshold: 10}]],
};
module.exports = config;Setting summaryThreshold: 1 will always show the summary of all failed tests.
By the way, one user missed coverage information in the printout. Note that it will be printed before the summary of all failed tests. That’s unfortunate, but there is nothing we can do at the moment, because summary reporter runs the last (i.e. just before coverage reporter).
It might make sense to have new 'all-failed-tests' reporter. It makes the option read better 'all-failed-tests', {threshold: 10}, but might complicate setting up reporters too. Hm.. actually it could be good idea:
/** @type {import('jest').Config} */
const config = {
// just drafting, this is not implemented!
reporters: [
'default',
['all-failed-tests', {threshold: 10}]
],
};
module.exports = config;What "new env variable" are you talking about?
Thanks for the clarification. Yet another reason to make it explicit right here on the issue itself what is the solution to the problem. It was not clear to me at all when I saw it closed, or after skimming through the linked PR. Thanks again.
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.