/kafka-prometheus-exporter

Primary LanguageGoApache License 2.0Apache-2.0

kafka-prometheus-consumer-exporter

Simple Prometheus exporter to export Metrics around Offsets & Consumergroup. Currently it exports three types of Metrrics:

  • kafka_broker_current_offset Current Offset at the Broker by Topic/Partion
  • kafka_topic_oldest_offset Oldest Offset still current in topic
  • kafka_topic_in_sync_replica Number of replicas which are inSync with the Topic/Partition leader.
  • kafka_topic_under_replicated Bool if all replicas are inSync by topic/partition.

If you use old consumer (which store their offset in zookeeper), these two Metrics about consumergroups and their offset are also aviable:

  • kafka_consumergroup_current_offset Current offset of every consumergroup by Topic/Partition
  • kafka_consumergroup_lag Approximate Lag of every consumergroup by Topic/Partition. Please note the Lag is not 100% Correct.

Build

  1. Ensure Dependecies with dep ensure
  2. Build go binary go build -o kafka_exporter main/cmd.go

Usage

Build the Sourcecode and start the binary with the correct command line options.

Example Output:

# HELP kafka_broker_current_offset Current Offset of a Broker at Topic/Partition
# TYPE kafka_broker_current_offset gauge
kafka_broker_current_offset{cluster="kafkaCluster",partition="0",topic="test"} 721271
kafka_broker_current_offset{cluster="kafkaCluster",partition="0",topic="test-verf"} 704461
# HELP kafka_consumergroup_current_offset Current Offset of a ConsumerGroup at Topic/Partition
# TYPE kafka_consumergroup_current_offset gauge
kafka_consumergroup_current_offset{cluster="kafkaCluster",consumergroup="console-consumer-13363",partition="0",topic="test"} 720453
kafka_consumergroup_current_offset{cluster="kafkaCluster",consumergroup="console-consumer-2265",partition="0",topic="test"} 16
kafka_consumergroup_current_offset{cluster="kafkaCluster",consumergroup="console-consumer-45639",partition="0",topic="test-verf"} 704461
kafka_consumergroup_current_offset{cluster="kafkaCluster",consumergroup="console-consumer-58950",partition="0",topic="test"} 380475
# HELP kafka_consumergroup_lag Current Approximate Lag of a ConsumerGroup at Topic/Partition
# TYPE kafka_consumergroup_lag gauge
kafka_consumergroup_lag{cluster="kafkaCluster",consumergroup="console-consumer-13363",partition="0",topic="test"} 818
kafka_consumergroup_lag{cluster="kafkaCluster",consumergroup="console-consumer-2265",partition="0",topic="test"} 721255
kafka_consumergroup_lag{cluster="kafkaCluster",consumergroup="console-consumer-45639",partition="0",topic="test-verf"} 0
kafka_consumergroup_lag{cluster="kafkaCluster",consumergroup="console-consumer-58950",partition="0",topic="test"} 340796

Command line Options:

Argument Description Default
listen-address The address on which to expose the web interface and generated Prometheus metrics. :8080
telemetry-path Path under which to expose metrics. /metrics
zookeeper-connect Zookeeper connection string localhost:2181
cluster-name Name of the Kafka cluster used in static label kafka-cluster
refresh-interval Seconds to sleep in between refreshes 15