Kminion stopped working with a Loadbalancer ?

[motilevy]

I have been running Kminion for quite sometime successfully with the following:

 environment:
      - KAFKA_BROKERS=10.202.42.34:9092
      - KAFKA_TLS_ENABLED=false 
      - KAFKA_SASL_ENABLED=false 

I've updated my image today and now Kminion would not connect to my cluster and I see these errors

{"level":"info","ts":"2022-08-19T23:19:30.744Z","logger":"main.kafka_service","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata","seed_brokers":"kafka.internal"}
{"level":"warn","ts":"2022-08-19T23:19:40.746Z","logger":"main.kgo_client","msg":"unable to open connection to broker","addr":"kafka.internal:9092","broker":"seed 0","err":"dial tcp: i/o timeout"}
{"level":"warn","ts":"2022-08-19T23:19:45.744Z","logger":"main.kgo_client","msg":"unable to open connection to broker","addr":"kafka.internal:9092","broker":"seed 0","err":"dial tcp: i/o timeout"}
{"level":"fatal","ts":"2022-08-19T23:19:45.745Z","logger":"main.kgo_client","msg":"failed to test connectivity to Kafka cluster","error":"failed to request metadata: unable to dial: dial tcp: i/o timeout"}

My cluster nodes are behind a Network Load Balancer.

I can connect to the cluster using nc

#>nc -vz kafka.internal 9092
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 10.0.112.190:9092.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

Dns returns :

#>host kafka.internal
kafka.internal has address 10.0.66.56
kafka.internal has address 10.0.112.190
kafka.internal has address 10.0.42.34

If I change the broker to any of the ip's above, it works like a charm

{"level":"info","ts":1660951414.3810258,"caller":"app/config.go:63","msg":"the env variable 'CONFIG_FILEPATH' is not set, therefore no YAML config will be loaded"}
{"level":"info","ts":"2022-08-19T23:23:34.381Z","logger":"main","msg":"started kminion","version":"sha-63d501b65003a848c909e21800c2a8c966fc6bdf"}
{"level":"info","ts":"2022-08-19T23:23:34.382Z","logger":"main.kafka_service","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata","seed_brokers":"10.0.110.160:9092"}
{"level":"info","ts":"2022-08-19T23:23:34.390Z","logger":"main.kafka_service","msg":"successfully connected to kafka cluster","advertised_broker_count":3,"topic_count":53,"controller_id":1402,"kafka_version":"v2.5"}
{"level":"info","ts":"2022-08-19T23:23:34.392Z","logger":"main","msg":"listening on address","listen_address":":8080"}

kminion                     | {"level":"info","ts":1660951995.993117,"caller":"app/config.go:63","msg":"the env variable 'CONFIG_FILEPATH' is not set, therefore no YAML config will be loaded"}
kminion                     | {"level":"info","ts":"2022-08-19T23:33:15.993Z","logger":"main","msg":"started kminion","version":"sha-63d501b65003a848c909e21800c2a8c966fc6bdf"}
kminion                     | {"level":"info","ts":"2022-08-19T23:33:15.994Z","logger":"main.kafka_service","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata","seed_brokers":"10.0.112.190:9092"}
kminion                     | {"level":"info","ts":"2022-08-19T23:33:16.101Z","logger":"main.kafka_service","msg":"successfully connected to kafka cluster","advertised_broker_count":3,"topic_count":53,"controller_id":1402,"kafka_version":"v2.5"}
kminion                     | {"level":"info","ts":"2022-08-19T23:33:16.102Z","logger":"main","msg":"listening on address","listen_address":":8080"}

kminion                     | {"level":"info","ts":1660952061.0930696,"caller":"app/config.go:63","msg":"the env variable 'CONFIG_FILEPATH' is not set, therefore no YAML config will be loaded"}
kminion                     | {"level":"info","ts":"2022-08-19T23:34:21.093Z","logger":"main","msg":"started kminion","version":"sha-63d501b65003a848c909e21800c2a8c966fc6bdf"}
kminion                     | {"level":"info","ts":"2022-08-19T23:34:21.094Z","logger":"main.kafka_service","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata","seed_brokers":"10.0.42.34:9092"}
kminion                     | {"level":"info","ts":"2022-08-19T23:34:21.108Z","logger":"main.kafka_service","msg":"successfully connected to kafka cluster","advertised_broker_count":3,"topic_count":53,"controller_id":1402,"kafka_version":"v2.5"}
kminion                     | {"level":"info","ts":"2022-08-19T23:34:21.112Z","logger":"main","msg":"listening on address","listen_address":":8080"}

This is the Id of the image that works "quay.io/cloudhut/kminion@sha256:4bf256727c239afa20448ff76841f328c52a9f1a90713a2edb107f3339e98768"

Any idea ?

[motilevy]

docker issues 😮‍💨