/kafka-summit-london-2023

Demo codebase for Kafka Summit London 2023 talk

Primary LanguageJava

Kafka Summit 2023 - Hope is not a course of action!

A practical deep dive into the Observability of Streaming Applications

Demo code base used during our Kafka Summit London 2023 talk. Speakers: Kosta Chuturkov & Tim van Baarsen

Project modules and applications

Applications Port Avro Topic(s) Description
spring-kafka-producer 8080 YES stock-quotes Simple producer of random stock quotes using Spring Kafka & Apache Avro.
spring-kafka-consumer 8082 YES stock-quotes Simple consumer of stock quotes using using Spring Kafka & Apache Avro.
spring-kafka-streams 8083 YES stock-quotes, stock-quotes-exchange-nyse, stock-quotes-exchange-nasdaq, stock-quotes-exchange-ams Simple Kafka Streams application using Spring Kafka & Apache Avro.
plain-kafka-consumer - YES stock-quotes-exchange-nyse, stock-quotes-exchange-nasdaq, stock-quotes-exchange-ams Simple 'plain' Kafka consumer application
shaky-downstream-service 7999 NO not using Kafka Simple Spring Boot (slow and shaky) Rest API used to build up some artificial consumer lag in the consumer.
Module Description
avro-model Holds the Avro schema for the Stock Quote including avro-maven-plugin to generate Java code based on the Avro Schema. This module is used by both the producer, consumer and Kafka streams application.

Note Confluent Schema Registry is running on port: 8081 using Docker see: docker-compose.yml.

Component Description Url Category
Grafana - Tempo Distributed tracing shown in Grafana http://localhost:3000/explore Tracing
Tempo Distributed tracing backend http://localhost:3200/status Tracing
Jaeger Distributed tracing backend http://localhost:16686/ Tracing
Prometheus Time series databasee (Metrics backend) http://localhost:9090 Metrics
Prometheus (status) http://localhost:9090/status Metrics
Conduktor admin@conduktor.io password: admin http://localhost UI
Schema Registry Confluent Schema Registry (AVRO) http://localhost:8081 Schema

Version

  • Confluent Kafka: 7.5.x
  • Confluent Schema Registry: 7.5.x
  • Java: 19
  • Spring Boot: 3.1.3
  • Spring for Apache Kafka: 3.0.10
  • Apache Avro: 1.11.x

Build

Mvn project

./mvnw clean install

Docker images for the applications

docker-compose -f docker-compose.yml  -f docker-compose-applications.yml build

Or run script:

./buildDockerImages.sh

Run the examples

Infrastructure only

docker-compose up -d

To shutdown:

docker-compose down -v

With Open Telemetry agent

docker-compose -f docker-compose.yml -f docker-compose-applications.yml up -d

To shutdown:

docker-compose -f docker-compose.yml -f docker-compose-applications.yml down -v

Run with Micrometer

docker-compose -f docker-compose.yml -f docker-compose-micrometer.yml up -d

To shutdown:

docker-compose -f docker-compose.yml -f docker-compose-micrometer.yml down -v

Confluent Control Center & Confluent Monitoring interceptors

./mvnw clean install -Pmonitoring-interceptors
docker-compose -f docker-compose.yml -f docker-compose-control-center.yml up -d
./mvnw -P monitoring-interceptors spring-boot:run -pl spring-kafka-producer -Dspring-boot.run.arguments="--spring.profiles.active=monitoring-interceptors"
./mvnw spring-boot:run -pl shaky-downstream-service
./mvnw -P monitoring-interceptors spring-boot:run -pl spring-kafka-consumer -Dspring-boot.run.arguments="--spring.profiles.active=monitoring-interceptors"

Confluent Control Center

To shutdown:

docker-compose -f docker-compose.yml -f docker-compose-control-center.yml down -v