kc-etcd

An example Kafka Connect source connector, ingesting changes from etcd

The goal of this project is not primarily to provide a production-ready connector for etcd, but rather to serve as an example for a complete yet simple Kafka Connect source connector, adhering to best practices -- such as supporting multiple tasks -- and serving as an example connector for learning purposes (e.g. on connector implementation and testing) and basis for explorations of related KIPs such as KIP-618 ("Exactly-Once Support for Source Connectors").

Usage

http PUT localhost:8083/connectors/test-connector/config < register-test.json

Build

This project requires OpenJDK 11 or later for its build. Apache Maven is used for the build. Run the following to build the project:

mvn clean verify

Testing With Docker Compose

The Docker Compose set-up in docker-compose.yml provides the following infrastructure for manual testing:

Apache Kafka and ZooKeeper
Kafka Connect with the etcd connector
Three etcd clusters: one with three nodes, two with one nodes each

Prepare the connector plug-in:

mvn clean verify

Start Apache Kafka, Kafka Connect, ZooKeeper, and etcd:

docker-compose up

http PUT localhost:8083/connectors/test-connector/config < register-test.json

Put something into one of the etcd clusters:

docker-compose exec etcd-a-1 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl put foo bar"

Consume events from Kafka:

docker-compose exec kafka /kafka/bin/kafka-console-consumer.sh \
    --bootstrap-server kafka:9092 \
    --from-beginning \
    --property print.key=true \
    --topic etcd-a

Apply equivalent steps for clusters/topics etcd-b and etcd-c.

Shut down:

docker-compose down

Useful Commands

Delete connector:

http DELETE http://localhost:8083/connectors/test-connector

List topics:

docker-compose exec kafka /kafka/bin/kafka-topics.sh --zookeeper zookeeper:2181 --list

License