kafka-streams-trial

Trying kafka-streams

Pre-requisites

Install jdk 8
Download and unzip Kafka 2.4 and run the following commands after cd to the kafka install directory

    bin/zookeeper-server-start.sh config/zookeeper.properties
    bin/kafka-server-start.sh config/server.properties.

Create 2 topics in Kafka using the following commands

    bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 6 --topic raw
    bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 6 --topic valid
    bin/kafka-topics.sh --list --bootstrap-server localhost:9092 //This is to see that the topics have been created

Running the code

Clone this repo and cd to the cloned folder
Run mvn clean package
Run mvn exec:java -Dexec.mainClass=com.binderror.Streaming
Add messages to the raw topic by issuing the following commands after cd to the kafka install directory

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic raw
Add the following json as input {"name": "user1", "address" : {"line1":"xzy", "line2":"jenga", "pin":"560068"}}

Use a console consumer on the valid topic to see the messages that have been processed by the streaming consumer.

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic valid --from-beginning

Understanding the source code

Streaming has the simple boilerplate code to read the stream from the input, process and stream to the output application.properties has the input and output topic names that can be configured. It defaults to raw (input topic) and valid (output topic) along with other configuration

Benchmarking the code on your workstation

Insert messages into the raw topic using the below command. The below command would insert 10 mil messages of 100 chars to the raw topic

bin/kafka-producer-perf-test.sh --topic raw --num-records 10000000 --record-size 100 --throughput 5000000 --producer-props bootstrap.servers=localhost:9092

Execute the program so that it starts streaming from raw to valid.
You can look at the lag for the consumer group (t1 by default, can be changed in application.properties) of the valid topic every 10 seconds (using watch command) by executing the following command

   bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group t1

Observations