blendle/zapdriver

Stacktrace error reporting

Dasio opened this issue · 3 comments

Dasio commented

When I log error using

config := zapdriver.NewProductionConfig()
logger, err := config.Build(zapdriver.WrapCore(
	zapdriver.ReportAllErrors(true),
	zapdriver.ServiceName("service"),
))
logger.Error("something happened", zap.Error(errors.New("my error"))

I can see error in "Error reporting', no stacktrace.
Stacktrace is in stacktrace field, but it is not recognized by GCP error reporting?

According to: https://cloud.google.com/error-reporting/reference/rest/v1beta1/projects.events/report#ReportedErrorEvent
it should be included in message.

@JeanMertz Would you be willing to fix this or accept a PR for this issue?

bouk commented

My understanding is that the stack trace should be appended to the error message with two newlines between them. This might have to be done with a custom Encoder

We have this issue too. Given a log like below the stacktrace is not reported in Error Reporting.

zap.Error("oops", zap.Error(detailedErr))

However, the logs contain this:

{
  "insertId": "aaaaaaaaaa",
  "jsonPayload": {
    "caller": "rpcs/some.go:161",
    "context": {
      "reportLocation": {
        "filePath": "/build/pkg/rpcs/some.go",
        "functionName": "github.com/redacted/redacted/pkg/rpcs.(*server).GetSomething",
        "lineNumber": "161"
      }
    },
    "error": "rpc error: code = Aborted desc = too much contention on these datastore entities. please try again. entity group key: Some/1L",
    "logging.googleapis.com/traceSampled": false,
    "message": "oops",
    "stacktrace": `github.com/redacted/redacted/pkg/rpcs.(*server).GetSomething
    /build/pkg/rpcs/some.go:161
  github.com/redacted/shared-proto/auth/gen/go._Server_GetSomething_Handler.func1
    /build/pkg/shared-proto/auth/gen/go/something_grpc.pb.go:118
  github.com/redacted/redacted/pkg/kit.instrumentCalls
    /build/pkg/kit/server.go:180
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.2.2/chain.go:25
  go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
    /go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.16.0/interceptor.go:333
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.2.2/chain.go:25
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.2.2/chain.go:34
  github.com/redacted/shared-proto/auth/gen/go._Server_GetSomething_Handler
    /build/pkg/shared-proto/auth/gen/go/something_grpc.pb.go:120
  google.golang.org/grpc.(*Server).processUnaryRPC
    /go/pkg/mod/google.golang.org/grpc@v1.50.0/server.go:1318
  google.golang.org/grpc.(*Server).handleStream
    /go/pkg/mod/google.golang.org/grpc@v1.50.0/server.go:1659
  google.golang.org/grpc.(*Server).serveStreams.func1.2
    /go/pkg/mod/google.golang.org/grpc@v1.50.0/server.go:955`,
  "timestamp": "2023-01-26T12:13:21.585164121Z"     
  }
}

Therefore, simply moving stacktrace to stack_trace does not solve the issue. StackDriver will only use stack_trace and ignore message:

docs:
You can specify more than one of those fields. If more than one of those fields is specified, then the order of evaluation is: stack_trace, then exception, and then message.

So maybe we should use a custom encoder and wrap the error inside the zap.Error with message and put that in message instead.