/securing-kafka-blog

Secure Kafka cluster (in a VM) for development and testing

Primary LanguagePuppet

Secure Kafka Cluster (VM for testing and development)

Table of Contents

Overview

Based on the instructions in the Confluent blog post Apache Kafka Security 101, this project provides a pre-configured virtual machine to run a secure Kafka cluster using the Confluent Platform.

This VM is intended for development and testing purposes, and is not meant for production use.

What's included in the VM

Usage

Starting the VM and the secure Kafka cluster

First, you must install two prerequisites on your local machine (e.g. your laptop):

Then you can launch the VM from your local machine:

# Clone this git repository
$ git clone https://github.com/confluentinc/securing-kafka-blog
$ cd securing-kafka-blog

# Start and provision the VM (this may take a few minutes).
# This step will boot the VM as well as install and configure
# Kafka, ZooKeeper, Kerberos, etc.
$ vagrant up

Once the VM is provisioned, the last step is to log into the VM and start ZooKeeper and Kafka with security enabled:

# Connect from your local machine to the VM via SSH
$ vagrant ssh default

# You will see the following prompt if you're sucessfully connected to the VM
[vagrant@kafka ~]$
# Start secure ZooKeeper and secure Kafka
[vagrant@kafka ~]$ sudo /usr/sbin/start-zk-and-kafka

The services that will now be running inside the VM include:

  • *:9093 -- secure Kafka broker (SSL)
  • *:9095 -- secure Kafka broker (SASL_SSL)
  • *:2181 -- secure Zookeeper instance

Your local machine (the host of the VM) cannot access these ports: Because the VM has no port forwarding configured yet (cf. Vagrantfile), you can only access Kafka or ZooKeeper from inside the VM. You cannot, however, directly access Kafka or ZooKeeper from your local machine.

Test-driving the secure Kafka cluster

You can use the example commands in Apache Kafka Security 101 to test-drive this environment.

Simple example:

#
# The following commands assume that you're connected to the VM!
# Run `vagrant ssh default` on your local machine if you are not connected yet.
#

# Create the Kafka topic `securing-kafka`
[vagrant@kafka ~]$ export KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf"
[vagrant@kafka ~]$ kafka-topics --create --topic securing-kafka \
                                --replication-factor 1 \
                                --partitions 3 \
                                --zookeeper localhost:2181

# Launch the console consumer to continuously read from the topic `securing-kafka`
# You may stop the consumer at any time by entering `Ctrl-C`.
[vagrant@kafka ~]$ kafka-console-consumer --bootstrap-server localhost:9093 \
                                          --topic securing-kafka \
                                          --new-consumer \
                                          --consumer.config /etc/kafka/consumer_ssl.properties \
                                          --from-beginning

# In another terminal:
# Launch the console producer to write some data to the topic `securing-kafka`.
# You can then enter input data by writing some line of text, followed by ENTER.
# Every line you enter will become the message value of a single Kafka message.
# You may stop the producer at any time by entering `Ctrl-C`.
[vagrant@kafka ~]$ kafka-console-producer --broker-list localhost:9093 \
                                          --topic securing-kafka \
                                          --producer.config /etc/kafka/producer_ssl.properties

# Now when you manually enter some data via the console producer,
# then your console consumer in the other terminal will show you
# the same data again.

Another example is to run a secure Kafka Streams application against the secure Kafka cluster in this VM:

  • SecureKafkaStreamsExample demonstrate how to write a secure stream processing application; the example includes step-by-step instructions on how it can be run against the secure Kafka cluster in this VM

Stopping the VM

Once you're done experimenting, you can stop the VM and thus the ZooKeeper and Kafka instances via:

# Run this command on your local machine (i.e. the host of the VM)
$ vagrant destroy

Troubleshooting

Configuration files

Main configuration files for both Kafka and ZooKeeper are stored under /etc/kafka.

Notably:

  • /etc/kafka/server.properties -- Kafka broker configuration file
  • /etc/kafka/zookeeper.properties -- ZooKeeper configuration file

Security related configuration files are also found under:

  • /etc/security/keytabs
  • /etc/security/tls
  • /etc/krb5.conf

Log files

Inside the VM you can find log files in the following directories:

  • Kafka: /var/log/kafka -- notably the server.log

Useful references