Producer and Consumer for external access of kafka service
Opened this issue · 6 comments
Anyone can help me around with the producer and consumer commands for externally accessing the kafka-service ?
Tried using this command from outside the kubernetes cluster to create a topic :-
bin/kafka-topics.sh --create --zookeeper zookeeper.kafka.svc.cluster.local:2181 --replication-factor 3 --partitions 3 --topic test
BUT , no success. Got this error.
Exception in thread "main" org.I0Itec.zkclient.exception.ZkException: Unable to connect to zookeeper.kafka.svc.cluster.local:2181 at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:72) at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1228) at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157) at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131) at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:79) at kafka.utils.ZkUtils$.apply(ZkUtils.scala:61) at kafka.admin.TopicCommand$.main(TopicCommand.scala:53) at kafka.admin.TopicCommand.main(TopicCommand.scala) Caused by: java.net.UnknownHostException: zookeeper.kafka.svc.cluster.local: Name or service not known at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) at java.net.InetAddress.getAllByName0(InetAddress.java:1276) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380) at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:70)
You won't get outside zookeeper access. Do administration from inside, and use the new kind of bootstrapping for outside clients.
"Do administration from inside" . Pardon ! But I didn't get you. Can you please be a little more informative.
Much appreciated :)
With new kafka versions you basically only need zookeeper access for topic maintenance etc. For everything else use "bootstrap servers", which works using the Outside services.
" --override advertised.listeners="INTERNAL_PLAINTEXT://${HOSTNAME}.broker.kafka.svc.cluster.local:9092,EXTERNAL_PLAINTEXT://$(eval wget -t3 -T2 -qO- http://169.254.169.254/latest/meta-data/public-hostname):9093" "
I have been asking a lot of questions .
But, What is this External Plaintext referring to ?
And could you help me with the command for kafka-producer ?
@ranjanshubham, if it helps, I applied to zookeeper the same logic applied to the brokers, i.e. I added a NodePort service as in /outside-services/outside-xxx.yml and changed the zookeeper/10zookeeper-config.yml accordingly so that labels were applied to the zk pods in order for the service selector to apply.
I couldn't tell you how orthodox or otherwise this approach is but I currently use zookeeper to retrieve the bootstrap broker list for my external clients so I need the service exposed.
I think this is too generic for this repo. It's a general Kafka question and I can only refer to for example the Kafka Definitive Guide book.