Test failure handling in report/terminal.go

[kardolus]

Was wondering if this case is ever executed when a test failure occurs.

From the few tests I performed with a spec-fork, it seems like there is a panic when a test fails, causing the case never to run, and the summary to be skipped.

The reason I'm looking into this is because I want to print the failure output in a different color. Sometimes in spec it's hard to see wether there were failing tests. I have been wrapping go test in a shell script in order to print a test-status message (failure or success based on exit code). Would be better to change things in spec though.

No rush at all, playing around with the fork and trying to fix it, but perhaps you have tips/ideas. I was thinking, in case the failures can never actually be counted, we may want to leave the failure count out of the summary and print the number of tests/skipped tests instead.

[sclevine]

Hi @kardolus,

That case is definitely executed when a test failure occurs:

package somepkg

import (
	"testing"

	"github.com/sclevine/spec"
	"github.com/sclevine/spec/report"
)

func TestFailure(t *testing.T) {
	spec.Run(t, "spec", func(t *testing.T, when spec.G, it spec.S) {
		it("should fail", func() {
			t.Fail()
		})
	}, spec.Report(report.Terminal{}))
}

opal:test stephen$ go test .
Suite: spec
Total: 1 | Focused: 0 | Pending: 0
x
Passed: 0 | Failed: 1 | Skipped: 0

--- FAIL: TestFailure (0.00s)
    --- FAIL: TestFailure/spec (0.00s)
        --- FAIL: TestFailure/spec/should_fail (0.00s)
FAIL
FAIL	github.com/sclevine/spec/test	0.007s

Can you link me to code that demonstrates what you're seeing? Maybe you're trying to use spec with Gomega and not registering the *testing.T correctly? (See #4.)

Also, FYI, @joefitzgerald has a colorized reporter available here: https://github.com/joefitzgerald/rainbow-reporter

[kardolus]

Thanks @sclevine - All I did was change this line on the php-cnb to Expect(f.Build.Layers).NotTo(test.HaveLaunchMetadata(.... Just so I had a test failure to play with.

Will look into rainbow-reporter and Gomega registration. Perhaps this (L24) is an incorrect way of registering *testing.T?

Just verified that spec works as expected without Gomega, inserting t.Error("bad stuff happened") in the same file does print the summary.

[sclevine]

Perhaps this (L24) is an incorrect way of registering *testing.T?

Yep, see #4 for more details.
I'd recommend this way, because it will let you run the specs in parallel as well:

func Test(t *testing.T) {
	spec.Run(t, "Example", testFoo, spec.Parallel(), spec.Report(report.Terminal{}))
}

func testFoo(t *testing.T, when spec.G, it spec.S) {
	var g *gomega.WithT
	it.Before(func() { g = NewWithT(t) })

	when("one test passes and one fails", func() {
		it("passes and is shown as a pass", func() {
			g.Expect(foo()).To(Equal(5))
		})

		it("fails and is shown as a pass", func() {
			g.Expect(foo()).To(Equal(6))
		})
	})
}

Alternatively, I personally prefer go-cmp to Gomega.

[kardolus]

Thanks! That works. Will look into go-cmp.