Struggling to try to connect to kafka as a consumer
Closed this issue · 12 comments
I've setup my openwifi deployment in my LAN and I have adopted an AP, and all the services seem to be running as expected as far as I can tell.
My goal is to be able to connect with my deployment via Kafka, I think as a consumer, to get telemetry/operational data.
So far I have added this ports key to my kafka block so I am able to try to access the kafka service from the LAN via port 9092.
kafka:
image: "docker.io/bitnami/kafka:${KAFKA_TAG}"
networks:
openwifi:
env_file:
- kafka.env
restart: unless-stopped
depends_on:
- zookeeper
volumes:
- kafka_data:/bitnami/kafka
ports:
- "9092:9092"
I also messed around a little bit with KAFKA_BROKERLIST=kafka:9092 in owgw.env. I tried setting it up as openwifi.wlan.local:9092 but it seemed to make no difference.
I'm not sure how to get past this error now. The e30c74984b44 gets replaced with a different mac-address-looking value every time. I restart the service which makes me think that it is using the mac address of the ephemeral container but I can't prove that for sure yet.
kafkacat -C -b 172.16.8.253 -t healthcheck
% ERROR: Local: Host resolution failure: e30c74984b44:9092/1001: Failed to resolve 'e30c74984b44:9092': Name or service not known (after 1ms in state CONNECT)
Any and all help is appreciated!
The secret is to get into the docker-compose network that is set up here.
So if you docker exec -it /bin/bash
you should be able to run:
kafka-console-consumer.sh --bootstrap-server kafka:9092 --from-beginning -topic healthcheck
Alternatively you can run kcat within the docker-compose created network like so:
docker run --net openwifi_openwifi -it edenhill/kcat:1.7.1 -C -b kafka:9092 -t healthcheck
run 'docker network list' to confirm the openwifi_openwifi network name.
I'm not sure I understand how to run the first command -- it just returns immediately:
ubuntu@openwifi:~/wlan-cloud-ucentral-deploy/docker-compose$ sudo docker-compose exec -it kafka /bin/bash
ubuntu@openwifi:~/wlan-cloud-ucentral-deploy/docker-compose$
The second command was more successful though. It did print out a lot of health check data. Thank you!! But I'm still searching for a way to get the same result from outside the ubuntu vm, do I need to proxy it or make a new service to proxy it or something? 🤔
Looks like github removed some important stuff there (had in angle brackets)...the command was:
docker exec -it ID /bin/bash
where ID is the docker id of kafka. Do docker ps | grep bitnami/kafka
Kafka can be a bit difficult when it comes to the broker address, but there should be way to do it.
This probably gives you some insights into what the issues are:
https://stackoverflow.com/questions/51630260/connect-to-kafka-running-in-docker
If you want to move kafka messages onto another kafka bus (externally accessible) you may want to look at kafka-exporters. I've not had much to do with those but I think they should work in theory.
Thanks, I tried a couple of things but so far I can't get the kafka data without being locally bridged in. I tried the kafka exporter route with the first kafka exporter I found but that refused to connect no matter what I do so far.
kafka-exporter:
image: danielqsj/kafka-exporter
command: ["--kafka.server=kafka:9092" ]
ports:
- "9308:9308"
networks:
openwifi:
depends_on:
- kafka
I also tried adding in these lines to kafka.env
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,PLAINTEXT_HOST://openwifi.wlan.local:29092
but now the kafka service fails to start (I see this over and over):
kafka_1 | kafka 20:23:13.05 Welcome to the Bitnami kafka container
kafka_1 | kafka 20:23:13.05 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-kafka
kafka_1 | kafka 20:23:13.05 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-kafka/issues
kafka_1 | kafka 20:23:13.06
kafka_1 | kafka 20:23:13.06 INFO ==> ** Starting Kafka setup **
kafka_1 | kafka 20:23:13.16 WARN ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
kafka_1 | kafka 20:23:13.18 INFO ==> Initializing Kafka...
kafka_1 | kafka 20:23:13.19 INFO ==> No injected configuration files found, creating default config files
kafka_1 | kafka 20:23:13.38 INFO ==> ** Kafka setup finished! **
kafka_1 |
kafka_1 | kafka 20:23:13.40 INFO ==> ** Starting Kafka **
kafka_1 | [2024-07-10 20:23:15,509] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
kafka_1 | [2024-07-10 20:23:16,853] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
kafka_1 | [2024-07-10 20:23:17,028] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
kafka_1 | java.lang.IllegalArgumentException: requirement failed: advertised.listeners listener names must be equal to or a subset of the ones defined in listeners. Found PLAINTEXT,PLAINTEXT_HOST. The valid options based on the current configuration are PLAINTEXT
kafka_1 | at scala.Predef$.require(Predef.scala:281)
kafka_1 | at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1937)
kafka_1 | at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1896)
kafka_1 | at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1389)
kafka_1 | at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:1327)
kafka_1 | at kafka.Kafka$.buildServer(Kafka.scala:67)
kafka_1 | at kafka.Kafka$.main(Kafka.scala:87)
kafka_1 | at kafka.Kafka.main(Kafka.scala)
I want to add:
I this in the logs for the kafka-exporter:
ubuntu@openwifi:~/wlan-cloud-ucentral-deploy/docker-compose$ sudo docker-compose logs -f kafka-exporter
Attaching to openwifi_kafka-exporter_1
kafka-exporter_1 | I0710 20:27:57.605717 1 kafka_exporter.go:804] Starting kafka_exporter (version=1.7.0, branch=master, revision=8973a8c5f32973ee454f8edac25d8f3f15c9a82d)
kafka-exporter_1 | F0710 20:27:58.448757 1 kafka_exporter.go:905] Error Init Kafka Client: kafka: client has run out of available brokers to talk to: dial tcp 192.168.16.3:9092: connect: connection refused
openwifi_kafka-exporter_1 exited with code 255
I modified my docker-compose.yml to this based on https://hub.docker.com/r/bitnami/kafka
kafka:
image: "docker.io/bitnami/kafka:${KAFKA_TAG}"
networks:
openwifi:
env_file:
- kafka.env
restart: unless-stopped
depends_on:
- zookeeper
volumes:
- kafka_data:/bitnami/kafka
ports:
- "9092:9092"
- "9094:9094"
environment:
- KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://0.0.0.0:9094
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,EXTERNAL://172.16.8.253:9094
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
I'm now able to get data from running
sudo docker run --net openwifi_openwifi -it edenhill/kcat:1.7.1 -C -b 172.16.8.253:9094 -t healthcheck
on the OWGW but
when I run almost the same command, minus the network bit, on another machine, I don't get data:
sudo docker run -it edenhill/kcat:1.7.1 -C -b 172.16.8.253:9204 -t healthcheck 1 :(
%3|1720645096.821|FAIL|rdkafka#consumer-1| [thrd:172.16.8.253:9204/bootstrap]: 172.16.8.253:9204/bootstrap: Connect to ipv4#172.16.8.253:9204 failed: Connection refused (after 0ms in state CONNECT)
%3|1720645097.821|FAIL|rdkafka#consumer-1| [thrd:172.16.8.253:9204/bootstrap]: 172.16.8.253:9204/bootstrap: Connect to ipv4#172.16.8.253:9204 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
% ERROR: Failed to query metadata for topic healthcheck: Local: Broker transport failure
I see traffic in tcpdump and the port is up in nmap:
nmap -p 9094 172.16.8.253 :)
Starting Nmap 7.80 ( https://nmap.org ) at 2024-07-10 15:02 MDT
Nmap scan report for openwifi.wlan.local (172.16.8.253)
Host is up (0.00059s latency).
PORT STATE SERVICE
9094/tcp open unknown
Nmap done: 1 IP address (1 host up) scanned in 0.03 seconds
Try changing the external listener address to 172.16.8.253 as well.
KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://172.16.8.253:9094
You're trying to bypass the docker networking stuff here, I'm not sure this is the best way to do that, but haven't tried myself.
Also, it may be possible to change the network to host network mode and that should bypass some of that. But not sure components will be happy with that configuration.
I think I may have messed up a port somewhere, I am getting data now!
henry-rgnets:certs % kafkacat -C -b 172.16.8.253:9094 -t healthcheck :)
% Reached end of topic healthcheck [0] at offset 64049
@carsten989 Thank you so much for your help!
If possible try to ask on the Slack general channel. Maybe somebody has already attempted this.
Now, hopefully I can enhance this solution into the right fit for OW. Hopefully exposing 1 more port in kafka should be the biggest networking change, think. I don't think changing the networking mode is necessary.
Information about this is in JIRA https://telecominfraproject.atlassian.net/browse/WIFI-13943