Multiple Kafka inputs - 7.x
robrankin opened this issue · 3 comments
robrankin commented
It appears that multiple Kafka inputs all use the last defined set of bootstrap_servers
, assuming the group coordinator
in the below log output is the same.
The 1st defined Kafka input is never used.
Removing either one of the defined inputs allows the remaining input to work correctly.
- Version: 7.0.0
- Operating System: CentOS Linux release 7.4.1708 (Core)
- Config File (if you have sensitive info, please remove it):
input {
kafka {
bootstrap_servers => "01.servicebus.windows.net:9093"
security_protocol => "SASL_SSL"
sasl_mechanism => "PLAIN"
jaas_path => "/etc/logstash/01-jaas.config"
decorate_events => true
group_id => "logstash-01-1"
client_id => "logstash-01"
topics => ["logstash"]
id => "logstash-kafka-01"
tags => ["01"]
}
kafka {
bootstrap_servers => "02.servicebus.windows.net:9093"
security_protocol => "SASL_SSL"
sasl_mechanism => "PLAIN"
jaas_path => "/etc/logstash/02-jaas.config"
decorate_events => true
group_id => "logstash-01-2"
client_id => "logstash-02"
topics => ["logstash"]
id => "logstash-kafka-02"
tags => ["02"]
}
}
- Sample Data:
[org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-en1bdedv02ehn-0, groupId=logstash-en1bdess01-2] Discovered group coordinator 02.servicebus.windows.net:9093 (id: 2147483647 rack: null)
[org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-en1bdess01ehn-0, groupId=logstash-en1bdess01-1] Discovered group coordinator 02.servicebus.windows.net:9093 (id: 2147483647 rack: null)
robrankin commented
In fact, "The 1st defined Kafka input is never used." doesn't appear to be correct after repeated restarting. Seems random.
vbohata commented
jaas_path is JVM global config. Can not be two different configs for the same Java process. To solve this I created following pull request: #313
yaauie commented
I believe this issue is resolved with documentation about the global nature of the jaas_path
setting and the introduction of sasl_jaas_config
.