Continuous restart of Kafka Produce Consume Test pod

[shubham-pandey95]

First of all, thank you very much for putting this repo together. I used this repo to setup Kafka and the only thing I changed was to introduce podAntiAffinity and provisioning GCE disks and PVC beforehand rather than dynamic volume provisioning. Zookeeper and Kafka pod installation is working fine but when trying to run the tests, the produce-consume pod is facing continuous restarts:

NAME                       READY   STATUS    RESTARTS   AGE
kafkacat-sz4vz             3/3     Running   0          20h
produce-consume-zcwlz      3/3     Running   336        20h
replication-config-wckl5   1/1     Running   0          20h

After a little debugging, I found the issue to be coming from the producer container and these are logs for the pod before getting killed:

kubectl logs -f produce-consume-zcwlz -c producer -n test-kafka --previous
++ date --iso-8601=ns -u
+ echo '--- start produce-consume-zcwlz 2019-09-27T18:01:37,693687068+00:00 ---'
+ ./bin/kafka-topics.sh --zookeeper zookeeper.kafka:2181 --describe --topic test-produce-consume
Topic:test-produce-consume	PartitionCount:3	ReplicationFactor:2	Configs:
	Topic: test-produce-consume	Partition: 0	Leader: 0	Replicas: 0,2	Isr: 0,2
	Topic: test-produce-consume	Partition: 1	Leader: 2	Replicas: 2,1	Isr: 2,1
	Topic: test-produce-consume	Partition: 2	Leader: 1	Replicas: 1,0	Isr: 1,0
+ tail -f /shared/produce.tmp
+ ./bin/kafka-console-producer.sh --broker-list bootstrap.kafka:9092 --topic test-produce-consume --producer-property acks=-1
/bin/bash: line 1:   324 Broken pipe             tail -f /shared/produce.tmp
       325 Killed                  | ./bin/kafka-console-producer.sh --broker-list $BOOTSTRAP --topic test-produce-consume --producer-property "acks=$ACK

I increased the resource limits for this container to 500m and 800Mi but this issue is still present. Can't find anything substantial in the zookeeper and kafka logs. Any idea what could be causing this? Or any metric that I should check?