blendle/zapdriver

Getting certain or all error level logs reported using Error Reporting

shashankmehra opened this issue · 5 comments

shashankmehra commented

I am trying to get (some) error level logs reported to Error Reporting service. It doesn't seem to be happening when I use this library. To get reported an error log entry must have a context object attached to it: https://cloud.google.com/error-reporting/docs/formatting-error-messages

Does this library support attaching such a context so that the error gets reported?

JeanMertz commented

Unfortunately, the logging structures used by Stackdriver differ based on some of their components/features.

In this case, the documentation you linked, states the following:

When logging error data from App Engine, Kubernetes Engine or Compute Engine, the only requirement is for the log entry to contain the full error message and stack trace. It should be logged as a multi-line textPayload or in the message field of jsonPayload (see the LogEntry reference).

Since this library outputs JSON format, you need to follow the part I highlighted from the quote.

The JSON format described on that page is for the "report" API, which is not what is used by this library, as that's less related to logging and more to error reporting:

When sending error data via the report API or when using custom logs, the following JSON structure must be used.

So:

I hope that helps clear things up for you.

shashankmehra commented

Using stackdriver's Reporting API would have required a lot of code change in all my projects. I was searching for a solution which integrated with logging by using a log format which generates an error report. (All my projects use a common zap logger).

The "reporting" log format specifies that adding the keys serviceContext and context (with source code details) to the json log would generate an error report.

I know that this library uses the LogEntry format, but I couldn't find one which uses the reporting format. But these two formats don't seem to clash with each other (there are no keys in common). So I have added error reporting support in my fork by adding ServiceContext() and ErrorReport() as zap fields, and allowing the logger to be configured in a way so that all logs with error level and above would generate an error report. The default behavior remains unchanged.

You can read the changes to the library API: https://github.com/karixtech/zapdriver/blob/error_reporting/README.md#using-error-reporting
And you can see the diff here: master...karixtech:error_reporting

I have already integrated this with one of my services, and error reporting is working. If you think this feature addition would be within the scope of this library, I would be happy to send a PR.

PS: I would have used the Reporting API if the logging method had not worked for me.

JeanMertz commented

Thanks for the extra details. Having this implemented does sound interesting.

One question: does this mean that you can keep using the default logging API and get error reporting as well?

shashankmehra commented

We currently use the Kubernetes with fluentd for logging and reporting. But from this comment it seems this is a stackdriver backend feature and not a part of fluentd plugin. So using the logging API should work.

From the docs for error reporting format:

The following payloads will be correctly processed by Error Reporting if sent to the report API or logged in LogEntry.jsonPayload

{
  "serviceContext": {"service": "worker"},
  "message": "Cannot process job: Missing attribute 'userId'",
  "context": {
    "reportLocation": {
      "filePath": "cleanup.js",
      "lineNumber": 42,
      "functionName": "processJob"
    }
  }
}

Which is the format I went with instead of putting the stacktrace in message field.

So adding these extra fields to LogEntry.jsonPayload would generate an error report.

JeanMertz commented

Alright. That sounds good 👌

I briefly looked at your changes, they look mostly okay to me. There are some things we might want to clean up a bit, but we can discuss that once you’ve created a PR.