Latency in kafka producer
Opened this issue · 3 comments
Hi guys,
I deployed kafka from https://github.com/Yolean/kubernetes-kafka/tree/v2.1.0 on k8s 1.7
I noticed that when my pod(kafka producer) wants to send a message to kafka broker it takes a lot of time (>1minute). Have any of you experienced such a thing? Or any idea what might be causing this?
This is a very generic kafka problem. I'd be very interested to hear if the setup in this repo causes problems in any way, but to do the analysis you probably have to consult Kafka docs or other Kafka resources. I recommend the book Kafka the Definitive Guide as it contains lots of insights on monitoring etc. It could of course be a Kubernetes problem instead. For example saturated nodes, slow volumes or pod networking issues.
Agree that it seems like a generic kafka issue not with this manifest itself. I was not able to figure out what the issue could be. Was trying to gather leads on where should I look. I can assure that we have sufficiently large nodes which are not resource starved. Also for volumes we are mounting gp1 SSD's from ec2 so should be good enough. I am using a rancher based cluster and installed kafka in a separate namespace. My pod is trying to access kafka from a different namespace. BTW if we connect to an independent kafka installation outside the cluster (on dedicated ec2 machines) it works fine.
What client (lib) are you using. Is there any debug loggin, such as problems connecting to brokers?