Transparent Kafka setup that you can grow with. Good for both experiments and production.
How to use:
- Good to know: you'll likely want to fork this repo. It prioritizes clarity over configurability, using plain manifests and .propeties files; no client side logic.
- Run a Kubernetes cluster, minikube or real.
- Quickstart: use the
kubectl applys below. - Have a look at addons, or the official forks:
- kubernetes-kafka-small for single-node clusters like Minikube.
- StreamingMicroservicesPlatform Like Confluent's platform quickstart but for Kubernetes.
- Join the discussion in issues and PRs.
No readable readme can properly introduce both Kafka and Kubernetes, but we think the combination of the two is a great backbone for microservices. Back when we read Newman we were beginners with both. Now we've read Kleppmann, Confluent and SRE and enjoy this "Streaming Platform" lock-in 😄.
We also think the plain-yaml approach of this project is easier to understand and evolve than helm charts.
Keep an eye on kubectl --namespace kafka get pods -w.
The goal is to provide Bootstrap servers: kafka-0.broker.kafka.svc.cluster.local:9092,kafka-1.broker.kafka.svc.cluster.local:9092,kafka-2.broker.kafka.svc.cluster.local:9092
`
Zookeeper at zookeeper.kafka.svc.cluster.local:2181.
The Kafka book recommends that Kafka has its own Zookeeper cluster with at least 5 instances.
kubectl apply -f ./zookeeper/
To support automatic migration in the face of availability zone unavailability we mix persistent and ephemeral storage.
kubectl apply -f ./
You might want to verify in logs that Kafka found its own DNS name(s) correctly. Look for records like:
kubectl -n kafka logs kafka-0 | grep "Registered broker"
# INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(kafka-0.broker.kafka.svc.cluster.local,9092,PLAINTEXT)
That's it. Just add business value 😉. For clients we tend to use librdkafka-based drivers like node-rdkafka. To use Kafka Connect and Kafka Streams you may want to take a look at our sample Dockerfiles. And don't forget the addons.
For clusters that enfoce RBAC there's a minimal set of policies in
kubectl apply -f rbac-namespace-default/
In production you likely want to manually set Reclaim Policy, or your data will be gone if the generated volume claims are deleted.
This can't be done in manifests, at least not until Kubernetes 1.8.
Tests are based on the kube-test concept.
Like the rest of this repo they have kubectl as the only local dependency.
kubectl apply -f test/
# Anything that isn't READY here is a failed test
kubectl get pods -l test-target=kafka,test-type=readiness -w --all-namespaces