BrighterCommand/Brighter

[Feature] Reduce noise from Librdkafka on non-fatal errors

iancooper opened this issue · 0 comments

Is your feature request related to a problem? Please describe.
Librdkafka raises both fatal and non-fatal errors to your producer/consumer. At this time we log these as an Error. This is noisy, and often relates to errors that Librdkafka is handling and we cannot intercept.

Describe the solution you'd like
When Librdkafka raises an error we should determine if it is fatal. If it is fatal, we should log as Fatal; if it is non-fatal we should log a Warning. This allows users of Brighter to decide if they want to see non-Fatal errors more easily and escalates a Fatal error, which might indicate that we have a zombie service if it does not crash.

Describe alternatives you've considered
The alternative would be to log Fatal as Error, but Librdkafka is doing a lot of error handling for us, so most Errors that are non-Fatal are coped with by Librdkafka and do not require a response.