Learning Kafka by playing. This repo currently follows the tutorial by Confluent.
Run Kafka and its dependencies:
docker-compose up
List existing topics:
docker-compose exec kafka /usr/bin/kafka-topics --list --zookeeper zookeeper:2181
Simulate message producer:
# you may change the value of parameters if needed.
docker-compose exec kafka /usr/bin/kafka-producer-perf-test \
--num-records 10000 \
--topic dummy.topic \
--producer-props bootstrap.servers=kafka:9092 \
--record-size 500 \
--throughput 20
Consume message:
docker-compose exec kafka /usr/bin/kafka-console-consumer \
--bootstrap-server kafka:9092 \
--topic dummy.topic \
--from-beginning
Reset offset:
docker-compose exec kafka /usr/bin/kafka-consumer-groups \
--bootstrap-server kafka:9092 --group mygroup \
--reset-offsets --to-earliest \
--all-topics --execute
Since Kafka's JMX port has been exposed, you can monitor Kafka metrics through the JMX metrics.
This repo uses Prometheus + Grafana for monitoring Kafka metrics by exporting JMX metrics via Prometheus' JMX Exporter plugin.
- Prometheus will be available in localhost:9090
- Grafana will be available in localhost:3000
This repo uses Kafka Connect from Debezium for simulating Kafka Connect.
Add connector to Kafka Connect:
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8083/connectors/ -d @./config/register-mysql.json