An application that records stats about consumer group offset commits and reports them as prometheus metrics.
This application is useful for determining how frequently each consumer group in your Kafka cluster is committing the offsets and, for the chattiest ones, it can calculate what the offset deltas are over time.
If you wish to build the project you'll need JDK 11.
We also recommend Docker for running the local Kafka and Zookeeper instance for manual testing purposes.
This application is packaged and designed to be deployed using a Docker image.
It's available on Docker Hub as farmdawgnation/kafka-hawk.
The application can be configured by environment variables for most cases.
BOOTSTRAP_SERVERS— (required) The bootstrap servers setting for hawkGROUP_ID- (optional) The group id, defaults to "hawk"SASL_CONFIG- (optional) The SASL config if neededCLUSTER_NAME- (optional) The custom name for this cluster (if any). Useful for situations where multiple instances of Hawk are deployed to monitor different clusters. Defaults to "kafka".JSON_LOGGING- (optional) If this environment variable is present, logback will configure itself to use Logstash compatible JSON logs.
The application also has an optional feature that can report on the offset commit deltas. You have to enable this per consumer group because it requires more resources to track the offset diffs in memory. It can be enabled with and configured with the following environment variables:
FEATURES_DELTAS_ENABLED- Set to "true" to enable the featureFEATURES_DELTAS_GROUPS- A comma separated list of groups to report deltas on
This tool was written by Matt Farmer who by day slings code and architecture for Mailchimp's Data Application Development Team.