"Kafkanetes"

Run Apache Kafka and Apache ZooKeeper on OpenShift v3.

Proof of concept; builds following architectures:

1 ZooKeeper pod <-> 1 Kafka pod
1 ZooKeeper pod <-> 2 Kafka pods
3 ZooKeeper pods <-> 1 Kafka pod
3 ZooKeeper pods <-> 2 Kafka pods

Jim Minter, 24/03/2016

Quick start

Prerequirement: ensure you have persistent storage available in OpenShift. If not, read Configuring Persistent Storage.

Clone repository

$ git clone https://github.com/jim-minter/kafkanetes.git

(Optionally) import templates into OpenShift (requires elevated privileges)

If you follow this step, as an alternative you can use the UI for all subsequent steps. If you omit this step, in all subsequent steps substitute $ oc process -f kafkanetes/foo.yaml | oc create -f - for $ oc new-app foo.

$ for i in kafkanetes/*.yaml; do sudo oc create -f $i -n openshift; done


1. Build the Kafkanetes image, containing RHEL, Java, Kafka and its distribution of Zookeeper
   ```bash
$ oc new-app kafkanetes-build
$ oc logs --follow build/kafkanetes-1

Deploy 3-pod Zookeeper

$ oc new-app kafkanetes-deploy-zk-3


1. Deploy 2-pod Kafka
   ```bash
$ oc new-app kafkanetes-deploy-kafka-2

Follow the Apache Kafka Documentation Quick Start

Deploy a debugging container and connect to it

$ oc new-app kafkanetes-debug $ oc rsh $(oc get pods -l deploymentconfig=kafkanetes-debug --template '{{range .items}}{{.metadata.name}}{{end}}')


1. Create a topic
   ```bash
bash-4.2$ bin/kafka-topics.sh --create --zookeeper kafkanetes-zk:2181 --replication-factor 1 --partitions 1 --topic test

List topics

bash-4.2$ bin/kafka-topics.sh --list --zookeeper kafkanetes-zk:2181


1. Send some messages
   ```bash
bash-4.2$ bin/kafka-console-producer.sh --broker-list kafkanetes-kafka:9092 --topic test 
foo
bar 
baz
^D

Receive some messages

bash-4.2$ bin/kafka-console-consumer.sh --zookeeper kafkanetes-zk:2181 --topic test --from-beginning


## Notes

* Known issue: with this setup, Kafka advertises itself using a non-qualified domain name, which means that it can only be accessed by clients in the same namespace.  Further customisation to changed the announced domain name/port and/or use NodePorts to enable external access should be fairly straightforward.

* The forthcoming Kubernetes ["Pet Set"](https://github.com/kubernetes/kubernetes/pull/18016) functionality should help normalise the templates for 3-pod ZooKeeper and 2-pod Kafka.