Kafka topic cleaner

The goal is to clean up unused topics in Kafka

Unused topics are topics that match the following three conditions:

Not internal that used by kafka components.
Topics have not been consumed for a while, that the consumers are already cleaned up by kafka, which by default is 7 days.
Topic that is empty

Environment variables

BOOTSTRAP_SERVERS: broker hosts
IS_DELETE_ENABLED: without setting this, this application does not delete any topic and only log the topics should be deleted

How to use

Pull from dockerhub

docker run --rm --name kafka-topic-cleaner --network=host -e BOOTSTRAP_SERVERS=localhost:29092 sendohchange/kafka-topic-cleaner

Build locally

mvn clean package
docker build -t kafka-topic-cleaner .
docker run --rm --name --network=host kafka-topic-cleaner -e BOOTSTRAP_SERVERS=localhost:29092 kafka-topic-cleaner or
docker run --rm --name --network=host kafka-topic-cleaner -e BOOTSTRAP_SERVERS=localhost:29092 -e IS_DELETE_ENABLED=true kafka-topic-cleaner

How to test locally

Use docker-compose and it creates some sample topics that demonstrates deletions

Why not python

To know which topics are/were consumed, the library must implement ListConsumerGroup API. Only one python library is up-to-date, developed by confluent, but this API does not implement ListConsumerGroup API yet. confluentinc/confluent-kafka-python#223