athulpg007/kafka-python

Kafka with Docker and Python Demo

Source: YouTube Video

Spin up Zookeeper and Kafka using docker-compose

docker-compose -f docker-compose.yml up -d

Check both Zookeeper and Kafka are up and running

docker ps

Enter Kafka container shell and create a new topic

docker exec -it kafka /bin/sh
cd opt/kafka_2.13-2.8.1/bin
kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic messages

Create a new Python project in Pycharm, install kafka-python

pip install kafka-python

Start Kafka producer

Open a PyCharm terminal. python producer.py

Start Kafka consumer

Open a different PyCharm terminal. python consumer.py

Spin down Zookeeper and Kafka

docker-compose down

Exercise: Topic with 2 partitions, 2 consumers

Video link: YouTube

## Spin up Zookeeper and Kafka using docker-compose

docker-compose -f docker-compose.yml up -d

## Check both Zookeeper and Kafka are up and running

docker ps

Enter Kafka container shell and create a new topic

docker exec -it kafka /bin/sh
cd opt/kafka_2.13-2.8.1/bin
kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 2 --topic registered_user_2

Start Kafka producer

Open a PyCharm terminal. python producer_2.py

Start Kafka consumer

python consumer_2.py (open in two terminals)