/kafka-playground

Learning Kafka by doing

Primary LanguageShellMIT LicenseMIT

kafka-playground

Learning Kafka by playing. This repo currently follows the tutorial by Confluent.

Usage

Run Kafka and its dependencies:

docker-compose up

List existing topics:

docker-compose exec kafka /usr/bin/kafka-topics --list --zookeeper zookeeper:2181

Simulate message producer:

# you may change the value of parameters if needed.
docker-compose exec kafka /usr/bin/kafka-producer-perf-test \
    --num-records 10000 \
    --topic dummy.topic \
    --producer-props bootstrap.servers=kafka:9092 \
    --record-size 500 \
    --throughput 20

Consume message:

docker-compose exec kafka /usr/bin/kafka-console-consumer \
    --bootstrap-server kafka:9092 \
    --topic dummy.topic \
    --from-beginning

Reset offset:

docker-compose exec kafka /usr/bin/kafka-consumer-groups \
    --bootstrap-server kafka:9092 --group mygroup \
    --reset-offsets --to-earliest \
    --all-topics --execute

Monitoring

Since Kafka's JMX port has been exposed, you can monitor Kafka metrics through the JMX metrics.

This repo uses Prometheus + Grafana for monitoring Kafka metrics by exporting JMX metrics via Prometheus' JMX Exporter plugin.

Kafka Connect

This repo uses Kafka Connect from Debezium for simulating Kafka Connect.

Add connector to Kafka Connect:

curl -i -X POST -H "Accept:application/json" -H  "Content-Type:application/json" http://localhost:8083/connectors/ -d @./config/register-mysql.json