/Kafka_for_begginers

Labs from Udemy course

Primary LanguageJava

Install Java

sudo apt update
sudo apt install openjdk-11-jdk
sudo vim /etc/environment
JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64"
source /etc/environment

Download Kafka

wget https://archive.apache.org/dist/kafka/3.0.0/kafka_2.13-3.0.0.tgz
tar xzf kafka_2.13-3.0.0.tgz
mv kafka_2.13-3.0.0 ~

Starting Kafka

In one terminal:

~/kafka_2.13-3.0.0/bin/zookeeper-server-start.sh ~/kafka_2.13-3.0.0/config/zookeeper.properties

In other terminal:

~/kafka_2.13-3.0.0/bin/kafka-server-start.sh ~/kafka_2.13-3.0.0/config/server.properties

Topics

Listing topics

~/kafka_2.13-3.0.0/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

Creating topics

~/kafka_2.13-3.0.0/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic first_topic --partitions 1 --replication-factor 1
~/kafka_2.13-3.0.0/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic second_topic --partitions 3 --replication-factor 1
~/kafka_2.13-3.0.0/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic third_topic --partitions 3 --replication-factor 2

this last one will not be created because there is only one broker

~/kafka_2.13-3.0.0/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic third_topic --partitions 3 --replication-factor 1

Describing topics

~/kafka_2.13-3.0.0/bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic first_topic

Deleting topics

~/kafka_2.13-3.0.0/bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic first_topic

Consumer and producer

Producer

Simple producer

 ~/kafka_2.13-3.0.0/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic first_topic

Producer with ack

 ~/kafka_2.13-3.0.0/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic first_topic acks=all

Producer with key separator

 ~/kafka_2.13-3.0.0/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic first_topic --property parse.key=true --property key.separator=:

Consumer

Simple consumer

 ~/kafka_2.13-3.0.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first_topic

Consumer from begginer

 ~/kafka_2.13-3.0.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first_topic --from-beginning

Consumer from begginer with key and value fromatter

 ~/kafka_2.13-3.0.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first_topic --from-beginning --formatter kafka.tools.DefaultMessageFormatter --property print.timestamp=true --property print.key=true --property print.value=true

Consumer with consumer group

 ~/kafka_2.13-3.0.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first_topic --group my-first-consumer-group

Consumer Group

List consumers group

  ~/kafka_2.13-3.0.0/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

Describing consumers group

  ~/kafka_2.13-3.0.0/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-first-consumer-group

Resetting Offsets

  ~/kafka_2.13-3.0.0/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-first-consumer-group --reset-offsets --to-earliest --execute --all-topics
  ~/kafka_2.13-3.0.0/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-first-consumer-group --reset-offsets --shift-by -2 --execute --all-topics