logstash-plugins/logstash-integration-kafka

Document how topics pattern regex happens at kafka clients instead of broker

jsvd opened this issue · 0 comments

jsvd commented

If a user sets topics_pattern => /regex/ what happens is that the kafka client used by this integration requests a full list of topics from the broken, then applies the regex locally.

This has two potential consequences:

  1. if the kafka broker has a lot of topics this operation can be very slow, especially if there are also many consumers
  2. if kafka broker has ACL enabled and the user only has WRITE/DESCRIBE permissions on the topics that match the regex, then the user will be surprised that authorization exceptions will be returned for topics that don't match the pattern (because the client request the full list).

This is known issue https://issues.apache.org/jira/browse/KAFKA-10717 and we should note this in the documentation to avoid surprises.