/kafka-ssl-docker

A Kafka docker container with SSL enabled.

Primary LanguageShellApache License 2.0Apache-2.0

License

The docker image is available on Docker Hub

Kafka broker with SSL enabled using Docker

Table of Contents

Installation (three ways)

a. Docker Compose

  1. Copy the below block into a file docker-compose.yml
version: "3"
services:
  kafka:
    image: "pardhu1212/kafka-ssl:0.1.0"
    ports:
      # SSL port
      - "9093:9093"
      # Plaintext port
      - "9094:9094"
      # zookeeper
      - "2181:2181"
    init: true
    environment:
      PASSWORD: "PAss$$123worD"
      # This can be domain name or IP address
      DOMAIN: "www.mywebsite.com"
    container_name: kafkassl
  1. Run the command
docker compose up -d

b. Existing docker image

  1. Pull the image:
docker pull pardhu1212/kafka-ssl:0.1.0
  1. Run the container using the command:
docker run --init -d -p 9093:9093 -p 9094:9094 --name=kafkassl -e PASSWORD=password DOMAIN=www.mywebsite.com pardhu1212/kafka-ssl

c. Building a local image

  1. Ruild the docker image using
docker build -t kafka-ssl-local
  1. Run the container using
docker run --init -d -p 9093:9093 -p 9094:9094 --name=kafkassl kafka-ssl-local
Keystore file

Access the generated key store file by using command docker cp kafkassl:/kafka_2.12-2.5.0/ssl/server.keystore.jks keystore.jks

Environment variables

Variable Default value Importance Description
PASSWORD password HIGH The password that will be used to create keystore file. Must be 8 or more characters.
DOMAIN www.mywebsite.com HIGH Domain name to be used while creating the certificate.
KAFKA_HOME /kafka_2.12-2.5.0/ LOW Directory where Kafka is installed inside the container.
KEY_STORE /kafka_2.12-2.5.0/ssl/server.keystore.jks LOW Keystore jks file path to be used inside docker container.

Purpose

  • The primary purpose of the project is to create a kafka container with SSL enabled.
  • The secondary goal of the project is to learn about kafka with SSL, docker commands and an important supervisor process called runit.

Description

Kafka with SSL

In the file prepStartup.sh we can notice different openssl and keytool commands. To understand what we are doing here, we need to have a basic understanding of how SSL works.

SSL

When a server is SSL enabled, it provides a certificate and the client validates it. When we browse for https://www.google.com, the Google server first responds with a certificate along with some details, Your browser has a list of certificates(in fact Certifcate Authorities) that it will trust. Since the Google's certificate is signed by a trustworthy Certifcate Authority(CA) like Verizon, your browser allows further connection.

Kafka SSL also works in a similar way. If you create a kafka broker (an equivalent of Google server), you want to make it SSL enabled, you have to provide a certificate. This certificate should be signed by a certificate authority. In the production use case, you have to create the certificate and mail it to an actual and trusted CA so that they will sign it. Then you can use this whenever a client tries to connect to you.

But how can we achieve this in a development scenario? Then you can create your own CA and sign your own certificate. The shell script does exactly that. It will create a certifcate, sign it with a self created CA and store them in a keystore file. We use Keytool(provided by Java) and Openssl to create them.

Certificate creation and signing

  • The content of the certificate file is encrypted using an algorithm. Most people use RSA and the same has been used here.
  • The validity of the certifcate needs to be specified. Here the validity is chosen as 365 days from the day of creation.
  • The most important details when creating the certificate is the Common Name(CN). Most clients who receive the SSL certifcate will verify the name of the domain with the CN of the certifcate. If it does not match, although the certificate is issued a trusted CA, the connection might be rejected.