kafka-idle-topics

Tooling to find topics that are not utilized in your Kafka cluster.

Ensure efficient usage of your topics in Confluent Cloud by discovering topics that aren’t being used.

kafka-idle-topics evaluates an idle topic through these criteria:

It has no Consumer Groups (including inactive consumer groups that have offsets committed).
It has no active production during the time of assessment (30 seconds while running by default).
It has no data within its partitions.

To download, go to Run for different ways of usage. It is also possible to download a release. The output will be written to a file in the same directory. The name of this file is configurable.

Usage

Usage of ./kafka-idle-topics:
  -allowList value
    	A comma delimited list of topics to evaluate. It also accepts a path to a file containing a list of topics.
  -bootstrap-servers string
    	Address to the target Kafka Cluster. Accepts multiple endpoints separated by a comma. Can be set using env variable KAFKA_BOOTSTRAP
  -disallowList value
    	A comma delimited list of topics to exclude from evaluation. It also accepts a path to a file containing a list of topics.
  -filename string
    	Custom filename for the output if needed. (default "idleTopics.txt")
  -hideInternalTopics
    	Hide internal topics from assessment.
  -hideTopicsPrefixes value
    	Disqualify provided prefixes from assessment. A comma delimited list. It also accepts a path to a file containing a list.
  -idleMinutes int
    	Amount of minutes a topic should be idle to report it. Can be set using env variable KAFKA_IDLE_MINUTES
  -kafkaSecurity string
    	Type of connection to attempt. Options: plain_tls, plain (no tls), tls (one-way), none. (default "none")
  -password string
    	Password in the PLAIN module. Can be set using env variable KAFKA_PASSWORD
  -productionAssessmentTimeMs int
    	Timeframe to assess active production. (default 30000)
  -skip string
    	Filtering to skip. Options are: production, consumption, storage. This can be a comma-delimited list.
  -username string
    	Username in the PLAIN module. Can be set using env variable KAFKA_USERNAME
  -version
    	Print the current version and exit

Security

kafka-idle-topics supports the following security set-ups:

plain_tls: SASL_SSL w/ PLAIN
plain: SASL_PLAINTEXT w/ PLAIN (No TLS)
tls: SSL (One-way TLS)
none: PLAINTEXT (Default)

To add certificates the tool can trust for TLS connections, ensure they are located in the subdirectories go seeks. Commonly it is /etc/ssl/certs/.

Run

You can run kafka-idle-topics in multiple configurations:

Docker

Example with Confluent Cloud:

docker run \
  -e KAFKA_BOOTSTRAP=<<BOOTSTRAP-SERVERS>>\
  -e KAFKA_USERNAME=<<API-KEY>>\
  -e KAFKA_PASSWORD=<<API-SECRET>>\
  abrahamleal/kafka-idle-topics:latest

Locally

Downloading with Go, example with Confluent Cloud:

go get github.com/abraham-leal/kafka-idle-topics/cmd/kafka-idle-topics
go install github.com/abraham-leal/kafka-idle-topics/cmd/kafka-idle-topics
$GOPATH/bin/kafka-idle-topics -bootstrap-servers <<bootstrap>> -username <<API-KEY>> -password <<API-SECRET>> -kafkaSecurity plain_tls

Build From Source

Git clone, example with Confluent Cloud:

git clone https://github.com/abraham-leal/kafka-idle-topics
cd kafka-idle-topics/
go build ./cmd/kafka-idle-topics/*
./kafka-idle-topics -bootstrap-servers <<bootstrap>> -username <<API-KEY>> -password <<API-SECRET>> -kafkaSecurity plain_tls

Filtering the evaluation

It is now possible to filter the topics which are evaluated. Setting -allowList or/and -disallowList flags will accept either a comma delimited string, or a file containing comma delimited entries for topic names. If specifying a file, make sure it has an extension (such as .txt). A topic specified in -disallowList and -allowList will be disallowed by default.

Additionally, you may set -hideTopicsPrefixes to hide custom prefixes from the evaluation.

Feature Requests / Issue Reporting

This repo tracks feature requests and issues through Github Issues. If you’d like to see something fixed that was not caught by testing, or you’d like to see a new feature, please feel free to file a Github issue in this repo, I’ll review and answer at best effort.

Additionally, if you’d like to contribute a fix/feature, please feel free to open a PR for review.