An example Kafka Connect source connector, ingesting changes from etcd
The goal of this project is not primarily to provide a production-ready connector for etcd, but rather to serve as an example for a complete yet simple Kafka Connect source connector, adhering to best practices -- such as supporting multiple tasks -- and serving as an example connector for learning purposes (e.g. on connector implementation and testing) and basis for explorations of related KIPs such as KIP-618 ("Exactly-Once Support for Source Connectors").
http PUT localhost:8083/connectors/test-connector/config < register-test.json
This project requires OpenJDK 11 or later for its build. Apache Maven is used for the build. Run the following to build the project:
mvn clean verify
The Docker Compose set-up in docker-compose.yml provides the following infrastructure for manual testing:
- Apache Kafka and ZooKeeper
- Kafka Connect with the etcd connector
- Three etcd clusters: one with three nodes, two with one nodes each
Prepare the connector plug-in:
mvn clean verify
Start Apache Kafka, Kafka Connect, ZooKeeper, and etcd:
docker-compose up
Register the connector:
http PUT localhost:8083/connectors/test-connector/config < register-test.json
Put something into one of the etcd clusters:
docker-compose exec etcd-a-1 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl put foo bar"
Consume events from Kafka:
docker-compose exec kafka /kafka/bin/kafka-console-consumer.sh \
--bootstrap-server kafka:9092 \
--from-beginning \
--property print.key=true \
--topic etcd-a
Apply equivalent steps for clusters/topics etcd-b and etcd-c.
Shut down:
docker-compose down
Delete connector:
http DELETE http://localhost:8083/connectors/test-connector
List topics:
docker-compose exec kafka /kafka/bin/kafka-topics.sh --zookeeper zookeeper:2181 --list
This code base is available under the Apache License, version 2.