zookeeper_kafka_example_setup

An example methodology for setting up a Kafka and Zookeeper system

##############################################################

Spin-up Zookeeper and Kafka broker using docker-compose

############################################################## The docker-compose file spins up a single Zookeeper and a single Kafka broker.

$ docker-compose -f docker-compose.yml pull $ docker-compose up -d

Create topic called 'JOSH_TOPIC_1'

$ docker exec zookeeper_kafka_example_setup_kafka_1 kafka-topics
--create
--topic JOSH_TOPIC_1
--zookeeper 127.0.0.1:32181
--partitions 1
--replication-factor 1

Start a console consumer on topic 'JOSH_TOPIC_1'

$ docker exec zookeeper_kafka_example_setup_kafka_1 kafka-console-consumer
--bootstrap-server 127.0.0.1:9093
--topic JOSH_TOPIC_1
--partition 0
--from-beginning

Create interactive console producer on topic 'JOSH_TOPIC_1'

$ docker exec -it zookeeper_kafka_example_setup_kafka_1 bash $ kafka-console-producer --topic JOSH_TOPIC_1 --broker-list 127.0.0.1:9093

Use 'kafka-python' script to send message to topic 'JOSH_TOPIC_1'

$ python3 producer_example.py

See: https://kafka-python.readthedocs.io/en/master/usage.html for more details.

#####################################

Set up without docker-compose

#####################################

Set up a zookeeper

$ docker pull confluentinc/cp-zookeeper:latest $ docker run --restart unless-stopped -d --net=host --name=zookeeper-1
-e ZOOKEEPER_SERVER_ID=1 -e ZOOKEEPER_CLIENT_PORT=32181
-e ZOOKEEPER_TICK_TIME=2000 -e ZOOKEEPER_INIT_LIMIT=5
-e ZOOKEEPER_SYNC_LIMIT=2
confluentinc/cp-zookeeper:latest

Set up a kafka broker.

$ docker pull confluentinc/cp-kafka:latest $ docker run -d --restart unless-stopped --net=host --name=kafka-1
-e KAFKA_ZOOKEEPER_CONNECT=127.0.0.1:32181
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9093
-e KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS=600000
-e KAFKA_BROKER_ID=1
confluentinc/cp-kafka:latest

Create a topic.

$ docker exec kafka-1 kafka-topics --create --topic JOSH_TOPIC_1
--zookeeper 127.0.0.1:32181 --partitions 3 --replication-factor 1

Check topics.

$ docker exec kafka-1 kafka-topics --describe --zookeeper 127.0.0.1:32181

The output will be like:

Topic:JOSH_TOPIC_1 PartitionCount:3 ReplicationFactor:1 Configs: Topic: JOSH_TOPIC_1 Partition: 0 Leader: 1 Replicas: 1 Isr: 1 Topic: JOSH_TOPIC_1 Partition: 1 Leader: 1 Replicas: 1 Isr: 1 Topic: JOSH_TOPIC_1 Partition: 2 Leader: 1 Replicas: 1 Isr: 1 Topic:__confluent.support.metrics PartitionCount:1 ReplicationFactor:1 Configs:retention.ms=31536000000 Topic: __confluent.support.metrics Partition: 0 Leader: 1 Replicas: 1 Isr: 1