Simple Kafka Environment for Kubernetes

This repository contains all the files necessary to deploy on Kubernetes a simple Kafka environment comprised of: Zookeeper, Kafka, Schema Registry and Kafka Connect. Continue reading this file for a local deployment guide.

Requirements

The requirements to deploy this kafka environment are:

A Kubernetes Cluster (I suggest Minikube for local deployments).
To use Minikube you'll also need Docker.
Kubectl to operate the kubernetes cluster.

Installation Guide

This guide will focus on local deployment of the kafka environment. To deploy on your cloud environment just skip the Minikube setup steps and don't forget to add auth configs to all deployments.

Minikube

i. minikube start (To start the Kubernetes Cluster)

ii. minikube dashboard --url (Optional - To launch a web application for you to have a visual environment to check the kubernetes cluster and deployment's health).

iii. eval $(minikube docker-env) (Optional - to use minikube's docker environment instead of local environment, so that image builds are available on minikube).
Deploy Applications - Respecting order of dependencies between deployments

i. Deploy Zookeeper

kubectl create -f zookeeper-deployment.yaml

kubectl create -f zookeeper-ip-service.yaml

ii. Deploy Kafka

kubectl create -f kafka/kafka-deployment.yaml

kubectl create -f kafka/kafka-ip-service.yaml

iii. Deploy Schema Registry

kubectl create -f schema-registry/schema-registry-deployment.yaml

kubectl create -f schema-registry/schema-registry-ip-service.yaml

iv. Deploy Kafka Connect

docker build -t kafka-connect-custom:1.0.0 kafka-connect

kubectl create -f kafka-connect/kafka-connect-deployment.yaml

kubectl create -f kafka-connect/kafka-connect-ip-service.yaml
Make applications accessible outside of the Kubernetes cluster

i. Kafka - kubectl port-forward services/kafka-cluster 9092:9092

ii. Schema Registry - kubectl port-forward service/kafka-schema-registry 8081:8081

iii. Kafka Connect - kubectl port-forward service/kafka-connect 8083:8083

iv. Zookeeper - kubectl port-forward service/zookeeper 2181:2181

To make the applications accessible I suggest the port-forward approach instead of creating ingress resources on the kubernetes cluster because there are some issues with Minikube/Docker networking that prevent ingresses from working properly in some environments (namely Apple Silicon).

Access to applications

The Environment is now up and running and applications should be accessible at:

Schema Registry - http://localhost:8081
Kafka Connect - http://localhost:8083
Kafka - localhost:9092
Zookeeper - localhost:2181

Obviously, you can interact with the applications programatically, but for ad-hoc operations, I suggest to download the Confluent CLI, since it has the resources to create topics, launch producers and consumers with schema support. It is specially helpful for quickly testing stuff.

Helpers

Confluent CLI commands examples:
Launch Producer with schema support - kafka-avro-console-producer --topic test.topic --bootstrap-server localhost:9092 --property schema.registry.url=http://localhost:8081 --property value.schema.id=1 --property key.schema.id=2 --property key.serializer=org.apache.kafka.common.serialization.StringSerializer --property parse.key=true --property key.separator=":"
Launch Consumer with schema support - kafka-avro-console-consumer --topic test.topic --bootstrap-server localhost:9092 --property schema.registry.url=http://localhost:8081 --property value.schema.id=1 --property key.schema.id=2 --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer --property print.key=true --property key.separator=":" --from-beginning
Create Topic - kafka-topics --create --topic test.topic --bootstrap-server localhost:9092
Schema registry request bodies examples:

Body to create a schema for a message value

Request URL: <SchemaRegistryUrl>/subjects/<topic-name>-value/versions

{
"schema":"{\"name\":\"event\",\"type\":\"record\",\"fields\":[{\"name\":\"status\",\"type\":\"string\"},{\"name\":\"created_at\",\"type\":\"string\"}]}",
"schemaType":"AVRO"
}

Body to create a schema for a message key

Request URL: <SchemaRegistryUrl>/subjects/<topic-name>-key/versions

{
"schema":"{\"type\":\"string\"}",
"schemaType":"AVRO"
}

Troubleshooting

In this repository, we are using Confluent's docker images made compatible with Apple Silicon.