Kerberos authentication failure

[pthalasta]

I'm trying to write data to Kafka broker authenticated with Kerberos with GSSAPI and SASL_SSL. When I start the logstash agent it errors out with Received Timeout from Kerberos but when i use kinit to get the ticket from kerberos, it works perfectly fine without any issues.

Logstash Config:

     input {
      file {
        path => "<path of log file to monitor>"
      }
    }
    output {
      kafka {
            topic_id => "<topic>"
            bootstrap_servers => "broker-list"
            compression_type => "snappy"
            jaas_path => "<path/to/jaas/config/file>"
            kerberos_config => "/etc/krb5.conf"
            sasl_kerberos_service_name => "kafka"
            security_protocol => "SASL_SSL"
            client_id => "<unique identifier of client>"
            ssl_truststore_location => "<ssl truststore in jks format>"
            ssl_truststore_password => "<password for truststore>"
        }
    }

JAAS config

KafkaClient {
     com.sun.security.auth.module.Krb5LoginModule required
     useKeyTab=true
     keyTab="logstash.service.keytab"
     storeKey=true
     useTicketCache=false
     serviceName="kafka"
     debug=true
     principal="logstash/domain@REALM";
    }

• Version: 7.10.1
• Java Version: openjdk version "11.0.9.1" 2020-11-04
• Operating System: Ubuntu 18.04.5 LTS

How do I fix this issue?

[gni-icdc]

I have nearly the same behaviour with the use of a schema registry with Avro messages.
Connection to schema registry ends with a 401 error, due to the fact that connectivity checks are done in common.rb.

When I comment the check_for_schema_registry_connectivity_and_subjects method, the connection is ok:

def check_schema_registry_parameters
  if @schema_registry_url
    check_for_schema_registry_conflicts
    @schema_registry_proxy_host, @schema_registry_proxy_port  = split_proxy_into_host_and_port(schema_registry_proxy)
    check_for_key_and_secret
    # check_for_schema_registry_connectivity_and_subjects
  end
end

Is it because connectivity check is done without Kerkeros authentication?

[fbaligand]

I agree with @gni-icdc.

The problem is here:
https://github.com/logstash-plugins/logstash-integration-kafka/blob/master/lib/logstash/plugin_mixins/common.rb#L58

The Kafka call here is done using Manticore Ruby API. While other Kafka calls are done through Java component that deals with Kerberos authentication.

Please fix it, this is a blocking issue for us.

[jsvd]

Hi folks..I'm investigating this and I'd like to confirm a few things.
The schema registry feature is only available on the input plugin (not the output), and is only enabled when the schema_registry_url setting is declared in the pipeline configuration.

The opening issue shows a kafka output configuration, so there seem to be two separate issues here?
@fbaligand and @gni-icdc can you confirm your scenario involves the kafka input AND setting the schema_registry_url?

[gni-icdc]

Hi João @jsvd , yes exactly! Our scenario occurs with the input plugin and with the schema_registry_url set in this input.

[jsvd]

Let's continue this conversation in #72, since it's not related to the original issue raised by @pthalasta

[robbavey]

I believe this problem should have been solved with #97. Please update to the latest (10.8.0) version of the kafka integration plugin

[fbaligand]

Great!
I will test it!

[fbaligand]

Given my tests with recent versions of the plugin, this issue is fixed and can be closed.