A practical deep dive into the Observability of Streaming Applications
Demo code base used during our Kafka Summit London 2023 talk. Speakers: Kosta Chuturkov & Tim van Baarsen
Applications | Port | Avro | Topic(s) | Description |
---|---|---|---|---|
spring-kafka-producer | 8080 | YES | stock-quotes |
Simple producer of random stock quotes using Spring Kafka & Apache Avro. |
spring-kafka-consumer | 8082 | YES | stock-quotes |
Simple consumer of stock quotes using using Spring Kafka & Apache Avro. |
spring-kafka-streams | 8083 | YES | stock-quotes , stock-quotes-exchange-nyse , stock-quotes-exchange-nasdaq , stock-quotes-exchange-ams |
Simple Kafka Streams application using Spring Kafka & Apache Avro. |
plain-kafka-consumer | - | YES | stock-quotes-exchange-nyse , stock-quotes-exchange-nasdaq , stock-quotes-exchange-ams |
Simple 'plain' Kafka consumer application |
shaky-downstream-service | 7999 | NO | not using Kafka | Simple Spring Boot (slow and shaky) Rest API used to build up some artificial consumer lag in the consumer. |
Module | Description |
---|---|
avro-model | Holds the Avro schema for the Stock Quote including avro-maven-plugin to generate Java code based on the Avro Schema. This module is used by both the producer, consumer and Kafka streams application. |
Note Confluent Schema Registry is running on port: 8081
using Docker see: docker-compose.yml.
Component | Description | Url | Category |
---|---|---|---|
Grafana - Tempo | Distributed tracing shown in Grafana | http://localhost:3000/explore | Tracing |
Tempo | Distributed tracing backend | http://localhost:3200/status | Tracing |
Jaeger | Distributed tracing backend | http://localhost:16686/ | Tracing |
Prometheus | Time series databasee (Metrics backend) | http://localhost:9090 | Metrics |
Prometheus (status) | http://localhost:9090/status | Metrics | |
Conduktor | admin@conduktor.io password: admin |
http://localhost | UI |
Schema Registry | Confluent Schema Registry (AVRO) | http://localhost:8081 | Schema |
- Confluent Kafka: 7.5.x
- Confluent Schema Registry: 7.5.x
- Java: 19
- Spring Boot: 3.1.3
- Spring for Apache Kafka: 3.0.10
- Apache Avro: 1.11.x
./mvnw clean install
docker-compose -f docker-compose.yml -f docker-compose-applications.yml build
Or run script:
./buildDockerImages.sh
docker-compose up -d
To shutdown:
docker-compose down -v
docker-compose -f docker-compose.yml -f docker-compose-applications.yml up -d
To shutdown:
docker-compose -f docker-compose.yml -f docker-compose-applications.yml down -v
docker-compose -f docker-compose.yml -f docker-compose-micrometer.yml up -d
To shutdown:
docker-compose -f docker-compose.yml -f docker-compose-micrometer.yml down -v
./mvnw clean install -Pmonitoring-interceptors
docker-compose -f docker-compose.yml -f docker-compose-control-center.yml up -d
./mvnw -P monitoring-interceptors spring-boot:run -pl spring-kafka-producer -Dspring-boot.run.arguments="--spring.profiles.active=monitoring-interceptors"
./mvnw spring-boot:run -pl shaky-downstream-service
./mvnw -P monitoring-interceptors spring-boot:run -pl spring-kafka-consumer -Dspring-boot.run.arguments="--spring.profiles.active=monitoring-interceptors"
To shutdown:
docker-compose -f docker-compose.yml -f docker-compose-control-center.yml down -v