Small tests for Kafka 2.1 consumer api and streaming api, showing how to send, pull and stream messages using Java 8.
Kafka 0.11 has some nice new features like exactly-once semantics and atomic writes using the new transaction API. Read more about these feature in this excellent article Exactly-once Semantics are Possible: Here’s How Kafka Does it.
In order to run these small examples you need a running Kafka broker on
localhost port 9092 with a topic called test-topic.
Start a Kafka broker locally by using Blacktop Kafka Docker image.
docker run -d \
--name kafka \
-p 9092:9092 \
-e KAFKA_ADVERTISED_HOST_NAME=localhost \
-e KAFKA_CREATE_TOPICS="test-topic:1:1,topic-a:1:1,topic-b:1:1" \
blacktop/kafka
To run the same on a server change the host name to reflect :
docker run -d \
--name kafka \
-p 9092:9092 \
-e KAFKA_ADVERTISED_HOST_NAME=er-ts-appqa04.os.eon.no \
-e KAFKA_CREATE_TOPICS="test-topic:1:1" \
blacktop/kafka
More info about Kafka can be found in the Kafka Documentation
When the Docker container has started you can start and stop as needed.
docker stop kafka
docker start kafka
docker ps -a (lists all docker containers, even those not running)
There are two consumer examples, they connect to the broker using different group.id / application.id so they will both receive all messages and they can be run in parallel.
- AdminClientTest, connects to broker and list topics and cluster info
- TestConsumer, connects to broker and pulls messages
- TestProducer, connects to broker and send x messages of y size
- StreamConsumer, connects to broker and streams messages