Basic setup: docker-compose.yml
Component | Port | API / UI | Description | Documentation |
---|---|---|---|---|
Kafka broker | 9092 | Single Node Kafka Cluster | https://kafka.apache.org/documentation/ | |
Conduktor Plaform | 80 | Conduktor | Conduktor | https://docs.conduktor.io/platform/ |
Confluent Schema Registry | 8081 | Schema Registry | Schema Registry for Avro Schemas | https://docs.confluent.io/platform/current/schema-registry/index.html |
Kafka connect: docker-compose-kafka-connect.yml
Component | Port | API / UI | Description | Documentation |
---|---|---|---|---|
Nginx | 8083 | Nginx | Loadbalancer for Kafka connect cluster | https://nginx.org/en/docs/ |
Kafka Connect - Worker 1 | 18083 | Kafka Connect - Worker 1 | 3 worker nodes cluster | https://docs.confluent.io/platform/current/connect/index.html |
Kafka Connect - Worker 2 | 28083 | Kafka Connect - Worker 2 | " | https://docs.confluent.io/platform/current/connect/index.html |
Kafka Connect - Worker 3 | 38083 | Kafka Connect - Worker 3 | " | https://docs.confluent.io/platform/current/connect/index.html |
Conduktor Proxy: docker-compose-conduktor-proxy.yml
Component | Port | Browser | Description | Documentation |
---|---|---|---|---|
Conduktor Proxy | 8888, 6969 - 6970 | Conduktor Proxy API | Proxy in front of Kafka cluster | https://docs.conduktor.io/platform/proxy |
Kafka console producer / consumer docker-compose-console-consumer-producer.yml
Component | Port | Description | Documentation |
---|---|---|---|
Kafka broker used as console producer | none | Container used only for the console producer | https://developer.confluent.io/tutorials/kafka-console-consumer-producer-basics/confluent.html |
Kafka broker used as console consumer | none | Container used only for the console consumer | https://developer.confluent.io/tutorials/kafka-console-consumer-producer-basics/confluent.html |
Up:
docker-compose up -d
Down:
docker-compose down -v
- admin:
- username:
admin@admin.io
- password:
admin
- username:
Up:
docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml up -d
Down:
docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml down -v
Up:
docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml up -d
Down:
docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml down -v
Start generating random data using Kafka Connect Data Gen:
IntelliJ Http Client - Kafka Connect API calls
or using curl
:
Create connector:
curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
"name": "datagen-stock-quotes",
"config": {
"connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
"kafka.topic": "stock-quotes",
"value.converter.schemas.enable": "true",
"schema.filename": "/kafka-connect/datagen/avro/StockQuote.avsc",
"schema.keyfield": "symbol",
"max.interval": 10,
"iterations": -1,
"tasks.max": 1
}
}'
Update configuration:
curl -X POST http://localhost:8083/connectors/datagen-stock-quotes/config -H "Content-Type: application/json" -d '{
"connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
"kafka.topic": "stock-quotes",
"value.converter.schemas.enable": "true",
"schema.filename": "/kafka-connect/datagen/avro/StockQuote.avsc",
"schema.keyfield": "symbol",
"max.interval": 10,
"iterations": -1,
"tasks.max": 3
}'
Up:
docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml -f docker-compose-conduktor-proxy.yml up -d
Down:
docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml -f docker-compose-conduktor-proxy.yml down -v
Up:
docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml -f docker-compose-conduktor-proxy.yml -f docker-compose-console-consumer-producer.yml up -d
Down:
docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml -f docker-compose-conduktor-proxy.yml -f docker-compose-console-consumer-producer.yml down -v
Create topic:
docker exec kafka-console-producer \
kafka-topics \
--bootstrap-server conduktor-proxy:6969 \
--create --if-not-exists --topic conduktor-test-topic
List the topics:
docker-compose exec kafka-console-producer \
kafka-topics \
--bootstrap-server conduktor-proxy:6969 \
--list
Producer to the topic:
echo 'Hello World' | docker-compose exec -T kafka-console-producer \
kafka-console-producer \
--bootstrap-server conduktor-proxy:6969 \
--topic conduktor-test-topic
Consume from the topic:
docker-compose exec kafka-console-consumer kafka-console-consumer \
--bootstrap-server conduktor-proxy:6969 \
--from-beginning \
--topic conduktor-test-topic
./mvnw clean install
./mvnw spring-boot:run -pl spring-kafka-consumer
- https://github.com/conduktor/kafka-stack-docker-compose
- https://github.com/conduktor/conduktor-platform/
- https://github.com/conduktor/conduktor-platform/blob/main/example-local/
- https://github.com/conduktor/conduktor-proxy-demos/tree/main/chaos
- https://www.conduktor.io/kafka/how-to-start-kafka-using-docker
- Documentation: https://docs.conduktor.io/platform/