bookingcom/carbonapi

CI actions succeed even though they contain errors. Unit tests fail.

grzkv opened this issue · 1 comments

grzkv commented

The CI action has multiple errors on the check step but it is marked as successful. See
https://github.com/bookingcom/carbonapi/runs/6575940793
and
https://github.com/bookingcom/carbonapi/actions/runs/2306617572 (annotations section).

grzkv commented

Here are some brief investigation results.

  • The test actually succeeds and go test return a 0. So, GitHub actions work perfectly well.
  • GitHub errors (see img below) are simply our outputs to stderr. This makes sense.
  • We log test cases to stderr

    carbonapi/pkg/parser/parser_test.go

    Line 437 in 657db88

    t.Logf("run case: go test -run 'TestParseExpr/%s'", regexp.QuoteMeta(tt.s))
    and additional info

    carbonapi/app/carbonzipper/http_handlers_test.go

    Line 928 in 0730dbc

    t.Log(w.Body.String())

    This is correctly interpreted as an error but it's not.
  • We needlessly run CI tests with -v. Because of this, we output logs to CI that include error messages These are not interpreted as errors by GitHub but still look like errors in the logs.

image