Kafka Data Cluster

Warning Because this repo is based upon VirtualBox which isn't available vor Apple Silicon based Macs, i have to deprecated this repo.

2023: there are test builds of VirtualBox for Apple Silicon, but so far it is not stable enough.

Content

In case you need a local cluster providing Kafka (with SSL and ACL) including a monitoring suite.

AWS Version

see AWS

Prerequisites

Vagrant (tested with 2.2.18)
VirtualBox (tested with 6.1.26)
Ansible (tested with 4.5.0)
The VMs take approx 15 GB of RAM, so you should have more than that.

⚠️ Vagrant might ask you for your admin password. The reason behind is, that vagrant-hostsupdater is used to have the vms available with their names in your network.

Init

git clone https://github.com/markush81/kafka-cluster.git
vagrant up

Cluster

The result if everything wents fine should be

Coordinates

Servers

IP	Hostname	Description	Settings
192.168.10.2	mon-1	running elk and metricbeat	4096 MB RAM
192.168.10.3	mon-2	running grafana, prometheus and metricbeat	2048 MB RAM
192.168.10.4	kafka-1	running a zookeeper, kafka broker and metricbeat	2048 MB RAM
192.168.10.5	kafka-2	running a zookeeper, kafka broker and metricbeat	2048 MB RAM
192.168.10.6	kafka-3	running a zookeeper, kafka broker and metricbeat	2048 MB RAM

Connections

Name
Zookeeper	kafka-1:2182,kafka-2:2182,kafka-3:2182
Kafka Brokers	kafka-1:9093,kafka-2:9093,kafka-3:9093
Kibana	http://mon-1:5601
Elasticsearch	http://mon-1:9200
Grafana	http://mon-2:3000
Prometheus	http://mon-2:9090

Monitoring

System Overview

Kibana Infrastructure

Kafka Overview

Gathered with Prometheus Agent from Kafka Brokers.

Kibana Metricbeat

Kibana Logs

Usage

Zookeeper

vagrant ssh kafka-1

zookeeper-shell.sh -zk-tls-config-file /vagrant/exchange/ssl-client/client-zookeeper-ssl.properties kafka-1:2182/

Connecting to kafka-1:2182/
...

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: zookeeper-1:2182,zookeeper-3:2182(CONNECTED) 0] ls /
[cluster, controller, controller_epoch, brokers, zookeeper, admin, isr_change_notification, consumers, config]
[zk: zookeeper-1:2182,zookeeper-3:2182(CONNECTED) 1]

Kafka

Topic Creation

vagrant ssh kafka-1

kafka-acls.sh --zk-tls-config-file /vagrant/exchange/ssl-client/client-zookeeper-ssl.properties --authorizer-properties zookeeper.connect=kafka-1:2182 --add --operation Create --cluster --allow-principal User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE

kafka-acls.sh --zk-tls-config-file /vagrant/exchange/ssl-client/client-zookeeper-ssl.properties --authorizer-properties zookeeper.connect=kafka-1:2182 --add --operation Create --topic '*' --allow-principal User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE

kafka-acls.sh --zk-tls-config-file /vagrant/exchange/ssl-client/client-zookeeper-ssl.properties --authorizer-properties zookeeper.connect=kafka-1:2182 --add --operation Describe --topic '*' --allow-principal User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE

kafka-acls.sh --zk-tls-config-file /vagrant/exchange/ssl-client/client-zookeeper-ssl.properties --authorizer-properties zookeeper.connect=kafka-1:2182 --add --operation DescribeConfigs --topic '*' --allow-principal User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE

kafka-topics.sh --command-config /vagrant/exchange/ssl-client/client-broker-ssl.properties --bootstrap-server kafka-1:9093 --create --replication-factor 1 --partitions 4 --topic sample

Created topic "sample".

kafka-topics.sh --command-config /vagrant/exchange/ssl-client/client-broker-ssl.properties --bootstrap-server kafka-1:9093 --topic sample --describe

Topic: sample	PartitionCount: 4	ReplicationFactor: 1	Configs: flush.ms=1000,segment.bytes=1073741824
	Topic: sample	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
	Topic: sample	Partition: 1	Leader: 2	Replicas: 2	Isr: 2
	Topic: sample	Partition: 2	Leader: 1	Replicas: 1	Isr: 1
	Topic: sample	Partition: 3	Leader: 0	Replicas: 0	Isr: 0

ACL for producers and consumers

kafka-acls.sh --zk-tls-config-file /vagrant/exchange/ssl-client/client-zookeeper-ssl.properties --authorizer-properties zookeeper.connect=kafka-1:2182 --add --producer --topic sample --allow-principal User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE

kafka-acls.sh --zk-tls-config-file /vagrant/exchange/ssl-client/client-zookeeper-ssl.properties --authorizer-properties zookeeper.connect=kafka-1:2182 --add --consumer --topic sample --allow-principal User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE  --group console --resource-pattern-type PREFIXED

kafka-acls.sh --zk-tls-config-file /vagrant/exchange/ssl-client/client-zookeeper-ssl.properties --authorizer-properties zookeeper.connect=kafka-1:2182 --list

Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=console, patternType=PREFIXED)`:
 	(principal=User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE, host=*, operation=READ, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=CLUSTER, name=kafka-cluster, patternType=LITERAL)`:
 	(principal=User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE, host=*, operation=CLUSTER_ACTION, permissionType=ALLOW)
	(principal=User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE, host=*, operation=CREATE, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=*, patternType=LITERAL)`:
 	(principal=User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE, host=*, operation=CREATE, permissionType=ALLOW)
	(principal=User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=__consumer_offsets, patternType=LITERAL)`:
 	(principal=User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE, host=*, operation=DESCRIBE, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=sample, patternType=PREFIXED)`:
 	(principal=User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE, host=*, operation=READ, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=sample, patternType=LITERAL)`:
 	(principal=User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE, host=*, operation=CREATE, permissionType=ALLOW)
	(principal=User:CN=kafka,OU=org,O=org,L=home,ST=Bavaria,C=DE, host=*, operation=DESCRIBE, permissionType=ALLOW)

Producer

kafka-console-producer.sh --broker-list kafka-1:9093,kafka-3:9093 --producer.config /vagrant/exchange/ssl-client/client-broker-ssl.properties --topic sample

Hey, is Kafka up and running?

Consumer

kafka-console-consumer.sh --bootstrap-server kafka-1:9093,kafka-3:9093 --consumer.config /vagrant/exchange/ssl-client/client-broker-ssl.properties  --group console-1 --topic sample --from-beginning

Hey, is Kafka up and running?

Producer Perf Test

kafka-producer-perf-test.sh --producer.config /vagrant/exchange/ssl-client/client-broker-ssl.properties --producer-props bootstrap.servers="kafka-1:9093,kafka-2:9093,kafka-3:9093" --topic sample --num-records 2000 --throughput 100 --record-size 256