sknsht/kafka-streams

Kafka Streams Demo Applications

Kafka Streams

As a first step to start all applications, you need to start the Zookeeper and the Kafka.

Step 1: Download the code

Download the 2.2.0 release and un-tar it.

Step 2: Start the Zookeeper server

> bin\windows\zookeeper-server-start.bat config\zookeeper.properties

Step 3: Start the Kafka server

> bin\windows\kafka-server-start.bat config\server.properties

WordCount

Implementation of WordCount algorithm, which returns a list of all unique words from the input text and their number of occurrences.

Step 1: Prepare the topics

Create the input topic named word-count-input and the output topic named word-count-output:

> bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 2 --topic word-count-input
> bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 2 --topic word-count-output

Step 2: Start the WordCount application

Start the console producer:

> bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic word-count-input

Start the console consumer in a separate terminal:

> bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic word-count-output --from-beginning --formatter kafka.tools.DefaultMessageFormatter --property print.key=true --property print.value=true --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer --property value.deserializer=org.apache.kafka.common.serialization.LongDeserializer

Run the WordCount java application.

Step 3: Process some data

Let's write a message with the console producer into the input topic word-count-input and check the output word count which will be written to the word-count-output topic and printed by the console consumer:

> bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic word-count-input
Hello world
This is Kafka Streams app

Favourite Colour

This application counts and displays the colours and the number of people who consider a particular colour as their favorite. It is assumed that people can change their preferences.

Step 1: Prepare the topics

Create the input topic named fav-colour-input and the output topic named fav-colour-output:

> bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 2 --topic fav-colour-input
> bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 2 --topic fav-colour-output

Step 2: Start the FavouriteColour application

Start the console producer:

> bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic fav-colour-input

Start the console consumer in a separate terminal:

> bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic fav-colour-output --from-beginning --formatter kafka.tools.DefaultMessageFormatter --property print.key=true --property print.value=true --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer --property value.deserializer=org.apache.kafka.common.serialization.LongDeserializer

Run the FavouriteColour java application.

Step 3: Process some data

As input, the application accepts name, colour pairs. Let's write a message with the console producer into the input topic fav-colour-input and check the output data which will be written to the fav-colour-output topic and printed by the console consumer:

> bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic fav-colour-input
ann, green
alex, blue
ann, blue

Bank Balance

Demonstration of the exactly once semantics on the example of bank transactions.

Step 1: Prepare the topics

Create a topic named bank-transactions in which the producer will write the generated transactions and the output topic named bank-balance:

> bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic bank-transactions
> bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic bank-balance

Step 2: Run the console consumer

Run the console consumer for the topic that displays the generated transactions and for the topic that displays the current balance:

> bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic bank-transactions --from-beginning
> bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic bank-balance --from-beginning --formatter kafka.tools.DefaultMessageFormatter --property print.key=true --property print.value=true --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer --property value.deserializer=org.apache.kafka.common.serialization.StringDeserializer

Step 3: Run the java Producer

The producer generates user transaction json objects that contain the name, time of the transaction, and random amount from -100 to 100.

Step 4: Run the BankBalance java application

The application generates a json object with the number of user transactions, current balance and time of the last transaction.