Separate behaviour for retriable and non-retriable exceptions
praseodym opened this issue · 2 comments
praseodym commented
Currently this plugin will retry producing events for any exception, even those that are not retriable (e.g. a SerializationException). By default there is no limit on the number of retries which causes these kind of failures to infinitely hang the entire Logstash pipeline.
Fortunately, the Kafka client library already provides a RetriableException base class for exceptions that are retriable (such as a TimeoutException). It makes a lot of sense to implement retry logic only for exceptions that inherit RetriableException and drop or send-to-DLQ events that are not.
roaksoax commented
Closing as this has been fixed in logstash-plugins/logstash-integration-kafka#29, in Logstash Kafka Integration plugin version 10.5.0