/springboot-kafka

Create Kafka consumer and producer using Spring Boot

Primary LanguageJava

Kafka Consumer and Producer using Spring Boot

This project built to demonstrate Kafka Consumer and Producer using Spring Boot. Also experiment with scaling consumer to handle slower consumer problem.

Run Locally

Clone the project

  git clone git@github.com:ruliarso/springboot-kafka.git

Start Kafka Cluster

  #Windows Powershell
  docker-compose-v1.exe -f .\kafka-cluster.yaml up

Create topic random-number with 3 partitions. Open kafka-ui

Start Producer and consumer

  #Windows Powershell
  docker-compose-v1.exe -f .\producer-consumer.yml up

Use buildkit to build docker images.

  # Windows Powershell for docker-compose build ... (additional!)
  $env:COMPOSE_DOCKER_CLI_BUILD = 1;$env:DOCKER_BUILDKIT = 1;docker-compose-v1.exe -f .\producer-consumer.yml build
  
  #Linux
  COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -f producer-consumer.yml build

Experiment with scaling consumer in consumer-group

  #Making consumer slower than producer. Edit producer-consumer.yml:
  MESSAGE_PROCESSING_TIME=2000
  #Try to scale up consumer. Number of consumer == number of partition
  docker-compose-v1.exe -f .\producer-consumer.yml up --scale sbconsumer=3

Related

Here are some related resources used in this project.