/sample-kafka-streams-performance

Primary LanguageJava

Sample Kafka Stream peformance/External call

The purpose of this repository is to showcase Kafka Streams

Getting Started

Start the sample by invoking the following shell scripts

./start.sh

The platform is comsposed by:

  • A Kafka cluster (1 ZK + 1 broker)
  • 1 producer sending a Kafka message with strings as key and value. The message backoff can be configured via the MESSAGE_BACKOFF_MS env var.
  • 2 Kafka Streams applications:

    • A vanilla stream app: the Topology is accesiblle at /topology and metrics at /metrics

      • The topology has 2 sub-topologies and 1 intermediate repartition topic.
      • A custom latency sensor (exposed in /metrics) measures the latency just after the record is read from the input topic and just before is sent to the output topic.

    • A String Boot + Kafka streams app: the Topology is accesiblle at /api/topology and metrics at /api/metrics

      • The topolology has 1 sub-topology
      • A custom latency sensor (exposed in /metrics) measures the latency just after the record is read from the input topic and just before is sent to the output topic.
      • The topology invoke a mock Machine Learning Service.
        • The service accept a value wait for a configured amount of milliseconds before returning the same value.
        • The service latency can be configured via API
          • /api/machine-learning/configuration gets the current configuration
          • api/machine-learning/configuration/duration/<value> sets the duration with ms
      • For demo purpose, the record polling configuration has been updated
        • max.poll.records: 20 (Default 1000)
        • max.poll.interval.ms: 6000 (Default 300000)
  • A Monitoring stack
    • Prometheus to scrape the 2 Kafka stream applications metrics
    • Kafka lag exported to get the 2 Kafka stream applications metrics lag
    • Grafana with:
      • Kafka Streams performance sample dashboards showing
        • The thread, tasks opertaions latency avg (in particular proccess/puncate)
        • The applications total rebalances
        • The thread, tasks, processor record-e2e-latency
      • Default kafka lag exporter default Consumer lag dashboard

Tears down

Stop the sample by invoking the following shell scripts

./stop.sh