This repo provides docker files, k8s manifests and OpenShift templates for Apache Kafka® Confluent Platform Community Edition. The reason why we're not using helm charts or the kafka operator at this time is because we like plain manifests which are more suitable for the learning phase.
This implementation was developed and tested on EKS with k8s 1.10.3, with tk8 made k8s 1.11.2 and OpenShift 3.5 and 3.7.
If one would like to get it working on k8s < 1.9.x, the apiVersion
for StatefulSets should be changed to apps/v1beta1
or apps/v1beta2
instead of apps/v1
.
To deploy the ephemeral version for development create the kafka-confluent-5 namespace and deploy the kafka-broker, zookeeper and the related services as follow:
$ kubectl create ns kafka-confluent-5
$ kubectl create -f https://raw.githubusercontent.com/kubernauts/kafka-confluent-platform/master/k8s/streaming-ephemeral.yaml -n kafka-confluent-5
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kafka-0 1/1 Running 0 1m
kafka-1 1/1 Running 0 1m
kafka-2 1/1 Running 0 1m
zoo-0 1/1 Running 0 1m
zoo-1 1/1 Running 0 1m
zoo-2 1/1 Running 0 1m
To deploy the persistent version with PVCs, create the kafka-confluent-5 namespace, create the storage class gp2 (for aws-ebs here) and deploy the kafka-broker, zookeeper and the related services with persitent support as follow:
Please adapt the zone in the storage class where you've your k8s cluster:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp2
annotations:
storageclass.kubernetes.io/is-default-class: "true"
labels:
kubernetes.io/cluster-service: "true"
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
zone: "us-east-1a"
And run the follwoing commads:
$ kubectl create ns kafka-confluent-5
$ kubectl create -f https://raw.githubusercontent.com/kubernauts/kafka-confluent-platform/master/k8s/storageclass-aws-ebs.yaml
$ kubectl create -f https://raw.githubusercontent.com/kubernauts/kafka-confluent-platform/master/k8s/streaming-persistent-aws-ebs.yaml -n kafka-confluent-5
For OpenShift 3.5 create a project "kafka-confluent-5" in OpenShift and upload the template through OpenShift Webcosole:
Deploy confluent test-client
kubectl create -f confluent-testclient-confluent-5.yaml
kubectl exec -it confluent-client -- /usr/bin/kafka-topics --zookeeper zoo --topic topic1 --create --partitions 30 --replication-factor 3
kubectl exec -it confluent-client -- /usr/bin/kafka-topics --zookeeper zookeeper:2181 --list
kubectl exec -it confluent-client -- /usr/bin/kafka-topics --zookeeper zookeeper:2181 --describe
kubectl exec -it confluent-client -- /usr/bin/kafka-console-producer --broker-list broker:9092 --topic topic1
kubectl exec -it confluent-client -- /usr/bin/kafka-console-consumer --bootstrap-server broker:9092 --topic topic1 --from-beginning
kubectl exec -it confluent-client -- /usr/bin/kafka-topics --zookeeper zookeeper:2181 --delete --topic topic1
Deploy Kafkacat:
kubectl create -f kafkacat.yml
Exec into the producer container and write messages to topic1:
$ k exec -it kafkacat-12345 -c producer /bin/bash
$ for i in `seq 1 100`; do echo "hello kafka world" | kafkacat -b kafka:9092 -t topic1; done
In another shell exec into the cosumer container and read messages:
$ k exec -it kafkacat-12345 -c consumer /bin/bash
# kafkacat -C -b kafka:9092 -t topic1
Make sure your k8s cluster is properly configured for Helm:
helm init
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
Clone the great implementation of Charles Martinot for the Yahoo Kafka Manager:
$ git clone https://github.com/MacTynow/kafka-manager-chart.git
$ cd kafka-manager-chart
Adapt values.yaml
useKafkaZookeeper: true
zkHosts: "zoo"
Install Yahoo Kafka Manager with Helm:
$ helm install .
Follow the instructions of the helm output, and access the GUI, provide "zoo" as the Cluster Zookeeper Hosts (the headless service).
Deploy Rancher on your k8s cluster with a single command:
kubectl create -f https://raw.githubusercontent.com/kubernauts/tk8/master/addons/rancher/master.yaml
Rancher will be up in 2 minutes, you can get the LB address through:
$ kubectl get svc -n cattle-system
Add a project in Rancher by navigating to "Projects/Namespaces" in Rancher UI, e.g. "confluent-kafka-5-project" and move the namespace "confluent-kafka-5" namespace to the newly created project "confluent-kafka-5-project". With this you can add new users to the project and have control!
Select your newly created project "confluent-kafka-5-project", navigate to Catalog Apps in the nice menu bar of Rancher and add Prometheus from Rancher's Library Catalog with the follwoing settings:
Expose Prometheus using Layer 7 Load Balancer: false
Prometheus Service Type: ClusterIP
Prometheus Persistent Volume StorageClass: gp2 (whic was created through the storageclass on EKS)
Create Persistent Volume for Prometheus: true
Expose Grafana using Layer 7 Load Balancer: false
Grafana Service Type: NodePort
Grafana Persistent Volume Enabled: true
Storage Class for Grafana: gp2
Expose Alertmanager using Layer 7 Load Balancer: false
Alertmanager Service Type: ClusterIP
Create Persistent Volume for Alertmanager: true
Alertmanager Persistent Volume StorageClass: gp2
And click on launch, after 2 minutes you shall get:
$ k get pods -n prometheus
NAME READY STATUS RESTARTS AGE
prometheus-alertmanager-6df98765f4-f79kh 2/2 Running 0 3m
prometheus-grafana-5bf7b6d949-pldn7 2/2 Running 0 3m
prometheus-kube-state-metrics-6584885ccf-l7tkm 1/1 Running 0 3m
prometheus-node-exporter-5qvjc 1/1 Running 0 3m
prometheus-node-exporter-bsp48 1/1 Running 0 3m
prometheus-node-exporter-f64cr 1/1 Running 0 3m
prometheus-node-exporter-r7t7c 1/1 Running 0 3m
prometheus-node-exporter-stdjg 1/1 Running 0 3m
prometheus-server-5959898967-s6hjd 2/2 Running 0 3m
Run something like this to access the Grafana Dashboard:
$ kubectlforward prometheus-grafana-5bf7b6d949-pldn7 3000
Import the kafka-overview_rev1.json and grafana-kafka-dashboard.json provided in the root of this repo as "Kafka Overview" Dashboard. You'll get a bunch of other very useful dashboard provided throught the Rancher installation.
To clean up your environment, run:
$ k delete all -l app=kafka -n kafka-confluent-5
$ k delete pvc --all
$ k delete ns kafka-confluent-5
https://github.com/edenhill/kafkacat
https://github.com/Yolean/kubernetes-kafka
https://docs.confluent.io/current/app-development/kafkacat-usage.html
https://hackernoon.com/a-blockchain-experiment-with-apache-kafka-97ee0ab6aefc
https://hackernoon.com/simple-chatops-with-kafka-grafana-prometheus-and-slack-764ece59e707
Join us on Kubernauts Slack Channel: