--hide-successes is displaying test headings and names but not the result ... sometimes

[jfischoff]

We are using the --hide-successes flag and after adding 976 test it stopped functioning correctly. Instead of only displaying failures, it is displaying the heading and test names. We see this on both OSX and FreeBSD. Unfortunately I don't have a test case I can easily show just yet.

Setting TERM=dumb causes the correct output to display.

[UnkindPartition]

Tasty uses ansi-terminal features to display the headings and the name of the current test while it's running, and then erase them if it hasn't failed. So it's possible that this may break under extreme conditions — perhaps when the amount of text to erase overflows the terminal or something.

It may be a while until I get around to looking into this myself, but patches are welcome.

[jfischoff]

Thanks that gives me something to go on

[UnkindPartition]

Did you manage to create a simple reproducible example?

Just a deep test tree is not enough to trigger this. Here's what I tried:

import Test.Tasty
import Test.Tasty.HUnit
import Data.Function

deepTree :: TestTree
deepTree = flip fix depth $ \rec n ->
  if n == 0
    then testCase "Test case" $ return ()
    else testGroup ("Test group " ++ show (depth+1-n)) [rec (n-1)]
  where depth = 100

main = defaultMain deepTree

[UnkindPartition]

Closing until there's some additional info.

[jfischoff]

👍

[ekmett-da]

I can duplicate the result here at Digital Asset with our test harness. I get 800+ headers of various tests and then the bodies of the failing ones. This happens even when I set the terminal wide-enough that the text doesn't overflow. We do have tests layered several levels deep if that helps narrow the cause.

I also don't have a nice contained example, however.

[UnkindPartition]

Would it be feasible for you to replace all test bodies with return () and see whether the problem persists? If it does, you could maybe share the test tree.

[ekmett-da]

The tests are spread across probably 2-dozen different projects aggregated into one large test harness, making that a fair bit more awkward than it sounds.

[UnkindPartition]

This is under some sort of unix, right?

Could you do the following:

1. Build tasty from the branch force-ansi
2. Redirect the output of ./test --hide-successes to a file
3. See if catting the file reproduces the problem
4. If so, scrape the sensitive information and share the file

[luc-tielen]

Hi,

I also came across this bug when working on a hobby project of mine.
I only have the error when I don't have my terminal full screen (for example only occupying right 50% of the screen).
Making it full screen makes the flag work again.
Also TERM=dumb seems to fix it also as mentioned above.

Some things I noticed:

1. cat after redirecting to file works fine.
2. OK does not appear in output (probably because it is hidden)
3. FAIL appears in a weird location (probably due to wrap around of line?)
   Screenshot for clarification:
   
   If you get this error, and then resize the width of the terminal, output gets weird too:
   
   Kind of looks like it keeps thinking width stays the same. Newlines are messed up too.

NOTE: I did NOT run with tasty compiled on the force-ansi branch, don't know how to integrate that with stack (yet) :).
Hope this helps, it's an annoying bug and would really appreciate it being fixed!

Seems to be related to width of terminal? (Try sliding it back and forth around the width FAIL ends up completely on the next line).

[UnkindPartition]

@luc-tielen could you post a reproducible example? Feel free to include only testGroups and trivial testCases that replicate your structure.

[luc-tielen]

Sure, here you go: https://github.com/luc-tielen/hspec-terminal-bug

You can test it by running make test.
This can reproduce it for me when I change integer in the forM_ to a value >= 83 + change size of terminal to one half of my screen.
Full screen it does not occur.
Hope this helps, feel free to play around with example code.

[UnkindPartition]

Yes, I can finally reproduce this — thanks!

[UnkindPartition]

So the cause of the problem is that, when line wrapping occurs, the ANSI API treats the wrapped line as two separate lines. This is a bit surprising to me because if I resize the terminal back to the full size, the two lines become one — so the terminal remains aware that the two rendered lines are a single logical line.

Currently, tasty assumes that what is output as a single line really is a single line from ANSI point of view to be able to return on that line and to erase it.

We cannot use saveCursor/restoreCursor because they only support a single position, whereas we need an unbounded stack of positions.

We could use getCursorPosition, but it will be almost as fragile. If, while a long-running test is executing, you resize the terminal, then all saved coordinates become wrong.

@mpilgrem maybe you know a way to store a stack of terminal positions and be able to go back up to any of those positions that is robust to text wrapping and resizing the terminal?

[luc-tielen]

I don't really know the implementation of the code, but are all results printed at end? Or as they come in?
If they all arrive at the end, why not filter out the output from successful tests if the flag is set?
It might even be possible as they come in..

Might be a lot easier than ANSI terminal magic.. something worth looking into?

[mpilgrem]

Nothing occurs to me so far. For Windows users, I found this blog entry that discusses under 'word wrap' how the native Windows console handles things (behind the scenes, 'this line has been wrapped here' codes are recorded in the screen buffer). In those consoles, 'Wrap text output on resize' is a layout option that a user can disable/enable and - I think, based on some searches - that the API does not expose what the user has chosen. I think that alone makes it near impossible to have an approach that is robust to terminal resizing.

[UnkindPartition]

@luc-tielen the results are printed as they come in, and then the successful ones are erased. This way you get immediate feedback about which test is executing (especially if it's taking too long). Here's how it looks.

If you don't need this immediate feedback, then exporting TERM=dumb seems like a decent solution — it will tell tasty that if though this looks like a termnal, tasty shouldn't attempt to use its ANSI features.

@mpilgrem thanks for looking into this!

[UnkindPartition]

Hm, the issue with TERM=dumb is that it also disables color. I should add a flag to only disable erasing.

[gromakovsky]

For me --hide-successes also doesn't work when hSupportsANSI stdout returns True. I've found this issue and it looks like now it's broken even more than it was before. Apparently consoleOutputHidingSuccesses erases less than it should. According to the comments above the issues happens only if terminal is not wide enough, but for me it now happens even in wide terminal.

Here is a tiny repo which reproduces the problem.

1. If I run this test with TERM=dumb, it doesn't print any names (neither group names nor test case names). This is the behavior I expect and I assume that without TERM=dumb it should behave the same way.
2. If I run it with --hide-successes in wide terminal, I will see only group names:

Tests
  q

3. If I run it with --hide-successes in narrow terminal, I will see names of individual tests as well:

Tests
  2+2=4:                                                                                                                                                       8 is even                                                                                                                                                    q
    3+3=6:  

Note that 8 is even test has very long name (see sources), so in narrow terminal it occupies two lines and the last line is erased.

I understand that dealing with narrow terminals is hard for the reasons described above in this thread, but --hide-successes seems to be broken for wide terminals as well and I guess it might be easier to fix.

[UnkindPartition]

All: there is now a fix/workaround in master: you can pass --ansi-tricks=false on the command line or set TASTY_ANSI_TRICKS=false in the environment to disable the ANSI tricks. Could you please try it and confirm if it works for you?