- Topics: a particular stream of data.
- Similar to a table in a database (without all the constraints).
- You can have as many topics as you want.
- A topic is identified by its name.
- Topics are split in partitions
- Each partition is ordered
- Each message within a partition gets an increamental id, called offset
- A Kafka cluster is composed of multiple brokers (servers)
- Each broker is identified with its ID (integer)
- Each broker contains certain topic partitions
- After connecting to any broker (called a bootstrap broker), you will be connected to the entire cluster
- A good number to get started is 3 brokers, but some big clusters have over 100 brokers
docker run --rm -it
-p 2181:2181 -p 3030:3030 -p 8081:8081
-p 8082:8082 -p 8083:8083 -p 9092:9092
-e ADV_HOST=127.0.0.1
landoop/fast-data-dev
docker run --rm -it
-p 2181:2181 -p 3030:3030 -p 8081:8081
-p 8082:8082 -p 8083:8083 -p 9092:9092
-e ADV_HOST=192.168.99.100
landoop/fast-data-dev
docker run --rm -it --net=host landoop/fast-data-dev bash
kafka-topics --zookeeper 127.0.0.1:2181 --create --topic first_topic --partitions 3 --replication-factor 1
kafka-topics --zookeeper 127.0.0.1:2181 --list
kafka-topics --zookeeper 127.0.01:2181 --topic second_topic --delete
kafka-topics --zookeeper 127.0.01:2181 --describe --topic first_topic
kafka-console-producer
kafka-console-producer --broker-list 127.0.0.1:9092 --topic first_topic
kafka-console-consumer
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic
kafka-console-producer --topic first_topic --broker-list 127.0.0.1:9092
root@fast-data-dev / $ kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --from-beginning
hello
exit
hi
new message
Here is a new message!
thai nguyen
Hello from producer
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --from-beginning --partition 0
hello
exit
root@fast-data-dev / $ kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --consumer-property group.id=mygroup1 --from-beginning
hello
exit
hi
new message
Here is a new message!
thai nguyen
Hello from producer