/kubernetes-kafka

Kafka cluster as Kubernetes StatefulSet, plain manifests and config

Primary LanguageShellApache License 2.0Apache-2.0

Kafka for Kubernetes

This community seeks to provide:

  • Production-worthy Kafka setup for persistent (domain- and ops-) data at small scale.
  • Operational knowledge, biased towards resilience over throughput, as Kubernetes manifest.
  • A platform for event-driven (streaming!) microservices design using Kubernetes.

To quote @arthurk:

thanks for creating and maintaining this Kubernetes files, they're up-to-date (unlike the kubernetes contrib files, don't require helm and work great!

Getting started

We suggest you apply -f manifests in the following order:

That'll give you client "bootstrap" bootstrap.kafka.svc.cluster.local:9092.

Fork

Our only dependency is kubectl. Not because we dislike Helm or Operators, but because we think plain manifests make it easier to collaborate. If you begin to rely on this kafka setup we recommend you fork, for example to edit broker config.

Version history

tag k8s ≥ highlights
4.x 1.9+ Kafka 1.1 dynamic config
v4.1 1.9+ Kafka 1.0.1 new default config
v3.2 1.9.4, 1.8.9, 1.7.14 Required for read-only ConfigMaps #162 #163 k8s #58720
v3.1 1.8 The painstaking path to min.insync.replicas=2
v3.0 1.8 Outside access, modern manifests, bootstrap.kafka
v2.1 1.5 Kafka 1.0, the init script concept
v2.0 1.5 addons
v1.0 1 Stateful? In Kubernetes? In 2016? Yes.

All available as releases.

Monitoring

Have a look at:

Outside (out-of-cluster) access

Available for:

Fewer than three nodes?

For minikube, youkube etc:

Stream...