/dc-kafka

Docker Compose of Kafka

Primary LanguageDockerfileGNU General Public License v3.0GPL-3.0

Docker Compose of Apache Kafka

A very simple docker-compose of Apache Kafka.

This repo is only used on my Raspberry Pi 4 with Ubuntu 20.04 (ARM64).

How to use this docker-compose.yml

Build the kafka image first

Create .env from .env.example:

foo@bar:~$ cp .env.example .env

Then edit it to fit your requirement.

Run the following command to build the image at the beginning:

foo@bar:~$ docker-compose build
...

This will build an image named grammyjiang/kafka:<tag>, and it can be checked:

foo@bar:~$ docker run --rm -it grammyjiang/kafka:2.6.0 kafka-server-start --version
[2020-11-25 21:57:58,713] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
2.6.0 (Commit:62abe01bee039651)
foo@bar:~$ docker run --rm -it grammyjiang/kafka:2.6.0 kafka-server-start --help
[2020-11-25 21:59:06,179] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
USAGE: java [options] KafkaServer server.properties [--override property=value]*
Option               Description
------               -----------
--override <String>  Optional property that should override values set in
                       server.properties file
--version            Print version information and exit.

Start Apache Kafka

Create the local environment files

Create zookeeper.env from zookeeper.env.example and kafka.env from kafka.env.example:

foo@bar:~$ cp zookeeper.env.example zookeeper.env
foo@bar:~$ cp kafka.env.example kafka.env

Then edit it to fit your requirement.

Start the service

foo@bar:~$ docker-compose up --detach
Creating network "dc-kafka_nw-kafka" with driver "bridge"
Creating dc-kafka-zookeeper ... done
Creating dc-kafka           ... done

Read the log messages

foo@bar:~$ docker-compose logs --follow
...

Check the status

foo@bar:~$ docker-compose ps
       Name                     Command               State                          Ports
------------------------------------------------------------------------------------------------------------------
dc-kafka             docker-entrypoint.sh kafka ...   Up      0.0.0.0:9092->9092/tcp
dc-kafka-zookeeper   /docker-entrypoint.sh zkSe ...   Up      0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp

Stop the service and remove the data

Stop the service

foo@bar:~$ docker-compose down --volumes
Stopping dc-kafka           ... done
Stopping dc-kafka-zookeeper ... done
Removing dc-kafka           ... done
Removing dc-kafka-zookeeper ... done
Removing network dc-kafka_nw-kafka

kafkacat

kafkacat is a generic non-JVM producer and consumer for Apache Kafka >=0.8, think of it as a netcat for Kafka.

foo@bar:~$ kafkacat -V
kafkacat - Apache Kafka producer and consumer tool
https://github.com/edenhill/kafkacat
Copyright (c) 2014-2019, Magnus Edenhill
Version 1.5.0 (JSON, librdkafka 1.2.1 builtin.features=gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,sasl_oauthbearer)

Check the metadata

foo@bar:~$ kafkacat -L -b localhost:9092
Metadata for all topics (from broker 0: localhost:9092/0):
 1 brokers:
  broker 0 at localhost:9092 (controller)
 0 topics:

Read the message

foo@bar:~$ kafkacat -b localhost:9092 -t quickstart-events
% Auto-selecting Consumer mode (use -P or -C to override)
This is my first event
% Reached end of topic quickstart-events [0] at offset 1

Reference