mfridman/tparse

tparse ignoring some test runs?

titpetric opened this issue · 13 comments

titpetric commented

I find that some files that are present in the JSON get omitted from tparse. Would you be able to tell me why some are omitted?

{"Time":"2022-08-17T18:20:05.260649294Z","Action":"fail","Package":"github.com/TykTechnologies/tyk/gateway","Test":"TestOldMockResponse/whitelist","Elapsed":1.42}

I'm using -all, -nocolor, -format markdown. I don't find TestOldMockResponse in the markdown output.

Source: https://tyk-automated-tests.cdn.si/log/1660759791.json
tparse output: https://tyk-automated-tests.cdn.si/log/1660759791.md
pandoc md2html: https://tyk-automated-tests.cdn.si/log/1660759791.html

mfridman commented

Curious, before looking into it a bit. Are you using a testing package, such as https://github.com/matryer/is ?

titpetric commented

Testify/assert, and/or matryer/is, it's a bit fragmented so we're possibly using both in different packages.

mfridman commented

I ask because I was seeing something similar in 2 separate projects, specifically with this package.

I filed an issue here:

matryer/is#47

Once I removed it, I had no issues seeing all the expected output.

titpetric commented

I'm not sure this applies. The JSON has TestOldMockResponse failures, the produced tparse output does not. We're not using matryer/is here, but we are using testify/assert. If we filter by Action==fail, we can find these test failures in json.

mfridman commented

Gotcha, if the entry is in the .json output then it should be captured.

I'll write a test case and get a repro.

Can you confirm the Go version, and the tparse version ?

titpetric commented

go 1.17, tparse@latest (go install-ed today).

mfridman commented

One more question, when you say omitted, do you mean omitted from the Failure output and/or the tests table (the one displayed when running -all) ?

There is usually special handling of panic output, so want to understand what you're expecting

CleanShot 2022-08-19 at 09 02 14@2x

titpetric commented

It's omitted from the failure output and the tests table, can't find it at all in the markdown output. Please check that linked .json in the issue and the generated .md with tparse. I'm using a makefile to generate a backlog of logs, as I'm using the JSON to track flaky test data and then link failures. The flaky test surfaces a bunch of these omitted failed tests, check the index page of that host if you like. https://tyk-automated-tests.cdn.si/ - all single test failures are candidates that possibly don't show up in the .md output.

.PHONY: all clean

all: $(shell ls *.json | sed -e 's/.json/.html/')

clean:
	rm -f *.html

%.html:
	cat $*.json | tparse -all -nocolor -format markdown > $*.md || true
	pandoc $*.md -f markdown -t html -s -o $@
mfridman commented

Thanks. There is a slightly different code path as soon as a panic is detected by tparse. I suspect I'll have to fix that, but even then you won't get all test output because in go panics are end of the world afaik.

There might be running tests and as soon as a panic occurs we'll have a bunch of pending events for those running tests we'll never receive. tparse doesn't have a complete representation of the test, and the only thing we know for certain is a panic happened.

Very early on I made an assumption that if a panic happens you probably don't care about other output as it'll be incomplete.

titpetric commented