/kafka-commands

KAFKA COMMANDS

Kafka Command line is needed to install these commands Run inside the bin directory of kafka

  1. Create a new topic:
    bash ./kafka-topics.sh --create --bootstrap-server kafka-ip:9092 --replication-factor 1 --partitions 500 --topic topicName

  2. Increase number of partitions
    bash ./kafka-topics.sh --bootstrap-server kafka-ip:9092 --alter --topic topicName --partitions 6000

  3. Describe Topic - Get info about topic
    bash ./kafka-topics.sh --describe --zookeeper kafka-ip:2181 --topic topicName

  4. Delete Topics
    bash ./kafka-run-class.sh kafka.admin.TopicCommand --delete --topic topicName --bootstrap-server kafka-ip:2181

  5. Producer - Enter command wait for '>' to appear, type/paste message to publish and press Enter
    bash ./kafka-console-producer.sh --broker-list kafka-ip:9092 --topic topicName

  6. Consumer - Starts from the latest message ie message being published after starting consumer
    bash ./kafka-console-consumer.sh --bootstrap-server kafka-ip:9092 --topic topicName

    Consumer - Fetch messages from beginning ie all available messages and keep consuming indefinately
    bash ./kafka-console-consumer.sh --bootstrap-server kafka-ip:9092 --topic topicName --from-beginning

    Consumer - Fetch messages from beginning and stop after consuming N messages
    bash ./kafka-console-consumer.sh --bootstrap-server kafka-ip:9092 --topic topicName --from-beginning --max-messages=N

  7. Get Lag - Command will generate a big table like structure containing information about cosumers including their IPs and lag on each partition
    bash ./kafka-consumer-groups.sh --bootstrap-server kafka-ip:9092 --describe --group <group-id>

    Get Lag - Command will generate a single number, containing sum of lag on all partitions
    bash ./kafka-consumer-groups.sh --bootstrap-server kafka-ip:9092 --describe --group <group-name> | grep -E "LAG|*" | awk '{SUM += $6} END { print SUM}'

  8. Reset offset for a particular consumer group - If you want to go to the start of messages on a topic for a particular consumer group(can replace 0 with any offset number) :
    bash ./kafka-consumer-groups.sh --bootstrap-server kafka-ip:9092 --group group-name --reset-offsets --to-offset 0 --topic topic-name --execute

    Reset offset for a particular consumer group - If you want to go to the end of messages on a topic for a particular consumer group(latest messages) :
    bash ./kafka-consumer-groups.sh --bootstrap-server kafka-ip:9092 --group group-name --reset-offsets --to-latest --topic topic-name --execute

  9. Delete a consumer group
    bash ./kafka-consumer-groups.sh --bootstrap-server kafka-ip:9092 --delete --group group-name