Yolean/kubernetes-kafka

Unable to access kafka brokers externally after using LoadBalancer

ishantanu opened this issue · 6 comments

ishantanu commented

Hi,

I tried exposing the individual Kafka brokers with NodePort and using LoadBalancer as well which created load balancers for each broker. But I am still not able to access them from outside.

Here's what I am seeing in logs:

org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
	at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:132)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:235)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:196)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:547)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:483)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:412)
	at kafka.network.Processor.poll(SocketServer.scala:575)
	at kafka.network.Processor.run(SocketServer.scala:492)
	at java.lang.Thread.run(Thread.java:748)
[2018-10-04 20:43:35,184] WARN [SocketServer brokerId=0] Unexpected error from /172.20.50.156; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
	at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:132)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:235)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:196)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:547)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:483)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:412)
	at kafka.network.Processor.poll(SocketServer.scala:575)
	at kafka.network.Processor.run(SocketServer.scala:492)
	at java.lang.Thread.run(Thread.java:748)

Also, is there any other way to expose individual Kafka brokers externally without a need to create a load balancer for each of the brokers?

solsson commented

I've never seen these errors. There are other ways, to be found in various issues. For example #187 comes to mind.

ishantanu commented

@solsson I tried changing InternalIP to ExternalIP but it's not working. I just want to be able to connect my services which are outside of k8s cluster to Kafka which is running on the k8s cluster.

solsson commented

This is a complicated topic and success factors depend a lot on your particular cluster setup. You'll have to dive into the reported errors.

ishantanu commented

I am not sure why I am getting an internalIP value in pod description after changing InternalIP to ExternalIP. Can you point to some issue with this?

solsson commented

There's no magic here. The current init.sh script in the applied config map takes effect whenever you restart a kafka pod.

  • 👍1
ishantanu commented

@solsson NVM, got the external access working with NodePorts and modifying security groups. Thanks for the help, closing the issue.