This repository contains the steps to enable TLS for Kafka and an example Scala application which demonstrates the TLS connection from Producer and Consumer.
All these steps are summarization of the official Kafka documentation that you can find here.
-
Create keypair for our CA
openssl req -new -x509 -keyout ca-key -out ca-cert -days 9999
This will create
ca-cert
andca-key
files which we will use at further steps. -
Generate SSL key and certificate for each Kafka broker First we need to create a truststore using
java keytool
and we need to import ourca-cert
into it.keytool -keystore kafka.broker0.truststore.jks -alias ca-cert -import -file ca-cert
Then we need to create a keystore.
keytool -keystore kafka.broker0.keystore.jks -alias broker0 -validity 9999 -genkey -keyalg RSA -ext SAN=dns:localhost
Here, at this point it is important to set
Subject Alternative Name (SAN)
as your host DNS to prevent someHost Name Verification
error. Also, when we run this command, it will ask usWhat is your first and last name?
which represent theCommon Name (CN)
. We need to set this property as we set inSAN
value. The other approach to preventHost Name Verification
error is settingssl.endpoint.identification.algorithm
field as an empty string insideserver.properties
file.ssl.endpoint.identification.algorithm=
Now, let's create our certificate signing request and sign it.
keytool -keystore kafka.broker0.keystore.jks -alias broker0 -certreq -file ca-request-broker0 openssl x509 -req -CA ca-cert -CAkey ca-key -in ca-request-broker0 -out ca-signed-broker0 -days 9999 -CAcreateserial
For the last step, import the signed certificate and CA into keystore.
keytool -keystore kafka.consumer.keystore.jks -alias ca-cert -import -file ca-cert keytool -keystore kafka.consumer.keystore.jks -alias consumer -import -file ca-signed-consumer
-
Configuring Kafka Brokers
-
Inside the
server.properties
file we need to add following configurations.ssl.client.auth=required ssl.keystore.location=<PATH_TO_kafka.broker0.keystore.jks> //for the example application it is under resources folder. ssl.keystore.password=<keystore_password> //for the example application it is 123456 ssl.key.password=<key_password> //for the example application it is 123456 ssl.truststore.location=<PATH_TO_kafka.broker0.truststore.jks> //for the example application it is under resources folder. ssl.truststore.password=<truststore_password> //for the example application it is 123456 ssl.protocol=TLSv1.2
-
Also we need to change
listeners
andlistener.security.protocol.map
properties as follow:listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093 // to complitely disable not secured way you can delete `PLAINTEXT://localhost:9092` part. listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
And run the broker with the
server.properties
configuration.
-
-
Configuring Kafka Clients
- See the example project. In the application we created different keystores and truststores following the above steps for each client(Producer and Consumer)