Readiness probe failed: connection refused
thomas-bouvier opened this issue · 5 comments
thomas-bouvier commented
Hi there!
I am having issues with the broker pods being unable to start properly. I am using minikube. Here are the instructions I followed:
- Starting a fresh minikube cluster
- Applying namespace
- Applying ./rbac-namespace-default
- Applying ./zookeeper
- Applying ./kafka
The three Kafka pods are stuck into a CrashLoopBackOff because of the readiness probe which seems to fail.
readinessProbe:
tcpSocket:
port: 9092
minikube is v1.6.2 and Kubernetes is v1.17.0.
Some logs:
kubectl get all -n kafka -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/kafka-0 0/1 Error 3 92s 172.17.0.11 minikube <none> <none>
pod/kafka-1 0/1 CrashLoopBackOff 2 92s 172.17.0.12 minikube <none> <none>
pod/kafka-2 0/1 CrashLoopBackOff 2 92s 172.17.0.13 minikube <none> <none>
pod/pzoo-0 1/1 Running 0 110s 172.17.0.6 minikube <none> <none>
pod/pzoo-1 1/1 Running 0 110s 172.17.0.9 minikube <none> <none>
pod/pzoo-2 1/1 Running 0 110s 172.17.0.7 minikube <none> <none>
pod/zoo-0 1/1 Running 0 110s 172.17.0.8 minikube <none> <none>
pod/zoo-1 1/1 Running 0 110s 172.17.0.10 minikube <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/bootstrap ClusterIP 10.96.10.14 <none> 9092/TCP 92s app=kafka
service/broker ClusterIP None <none> 9092/TCP 92s app=kafka
service/pzoo ClusterIP None <none> 2888/TCP,3888/TCP 110s app=zookeeper,storage=persistent
service/zoo ClusterIP None <none> 2888/TCP,3888/TCP 110s app=zookeeper,storage=persistent-regional
service/zookeeper ClusterIP 10.96.90.13 <none> 2181/TCP 110s app=zookeeper
NAME READY AGE CONTAINERS IMAGES
statefulset.apps/kafka 0/3 92s broker solsson/kafka:2.3.0@sha256:b59603a8c0645f792fb54e9571500e975206352a021d6a116b110945ca6c3a1d
statefulset.apps/pzoo 3/3 110s zookeeper solsson/kafka:2.3.0@sha256:b59603a8c0645f792fb54e9571500e975206352a021d6a116b110945ca6c3a1d
statefulset.apps/zoo 2/2 110s zookeeper solsson/kafka:2.3.0@sha256:b59603a8c0645f792fb54e9571500e975206352a021d6a116b110945ca6c3a1d
Describe on one of the pods (kafka-0):
kubectl describe pod kafka-0 -n kafka
Name: kafka-0
Namespace: kafka
Priority: 0
Node: minikube/192.168.64.8
Start Time: Fri, 17 Jan 2020 14:24:14 +0100
Labels: app=kafka
controller-revision-hash=kafka-66874b96bc
kafka-broker-id=0
statefulset.kubernetes.io/pod-name=kafka-0
Annotations: kafka-listener-outside-host: 192.168.64.8
kafka-listener-outside-port: 32400
Status: Running
IP: 172.17.0.11
IPs:
IP: 172.17.0.11
Controlled By: StatefulSet/kafka
Init Containers:
init-config:
Container ID: docker://bb7f0c76d0b8c64db8887e845ea0311dfa085e46c063b55ed0b01176370e34c5
Image: solsson/kafka-initutils@sha256:f6d9850c6c3ad5ecc35e717308fddb47daffbde18eb93e98e031128fe8b899ef
Image ID: docker-pullable://solsson/kafka-initutils@sha256:f6d9850c6c3ad5ecc35e717308fddb47daffbde18eb93e98e031128fe8b899ef
Port: <none>
Host Port: <none>
Command:
/bin/bash
/etc/kafka-configmap/init.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 17 Jan 2020 14:24:15 +0100
Finished: Fri, 17 Jan 2020 14:24:17 +0100
Ready: True
Restart Count: 0
Environment:
NODE_NAME: (v1:spec.nodeName)
POD_NAME: kafka-0 (v1:metadata.name)
POD_NAMESPACE: kafka (v1:metadata.namespace)
Mounts:
/etc/kafka from config (rw)
/etc/kafka-configmap from configmap (rw)
/opt/kafka/libs/extensions from extensions (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-t99db (ro)
Containers:
broker:
Container ID: docker://f16e7b9ae820b905071a853e7c926d2e174c19ce5ba7f6a6022d353ed96e0e77
Image: solsson/kafka:2.3.0@sha256:b59603a8c0645f792fb54e9571500e975206352a021d6a116b110945ca6c3a1d
Image ID: docker-pullable://solsson/kafka@sha256:b59603a8c0645f792fb54e9571500e975206352a021d6a116b110945ca6c3a1d
Ports: 9092/TCP, 9094/TCP, 5555/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
./bin/kafka-server-start.sh
/etc/kafka/server.properties
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Fri, 17 Jan 2020 14:28:03 +0100
Finished: Fri, 17 Jan 2020 14:28:31 +0100
Ready: False
Restart Count: 5
Limits:
memory: 600Mi
Requests:
cpu: 100m
memory: 100Mi
Readiness: tcp-socket :9092 delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
CLASSPATH: /opt/kafka/libs/extensions/*
KAFKA_LOG4J_OPTS: -Dlog4j.configuration=file:/etc/kafka/log4j.properties
JMX_PORT: 5555
Mounts:
/etc/kafka from config (rw)
/opt/kafka/libs/extensions from extensions (rw)
/var/lib/kafka/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-t99db (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-kafka-0
ReadOnly: false
configmap:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: broker-config
Optional: false
config:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
extensions:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
default-token-t99db:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-t99db
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 5m33s (x2 over 5m34s) default-scheduler error while running "VolumeBinding" filter plugin for pod "kafka-0": pod has unbound immediate PersistentVolumeClaims
Normal Scheduled 5m30s default-scheduler Successfully assigned kafka/kafka-0 to minikube
Normal Pulled 5m29s kubelet, minikube Container image "solsson/kafka-initutils@sha256:f6d9850c6c3ad5ecc35e717308fddb47daffbde18eb93e98e031128fe8b899ef" already present on machine
Normal Created 5m29s kubelet, minikube Created container init-config
Normal Started 5m29s kubelet, minikube Started container init-config
Warning Unhealthy 4m44s (x2 over 5m4s) kubelet, minikube Readiness probe failed: dial tcp 172.17.0.11:9092: connect: connection refused
Normal Pulled 3m23s (x5 over 5m27s) kubelet, minikube Container image "solsson/kafka:2.3.0@sha256:b59603a8c0645f792fb54e9571500e975206352a021d6a116b110945ca6c3a1d" already present on machine
Normal Created 3m23s (x5 over 5m27s) kubelet, minikube Created container broker
Normal Started 3m23s (x5 over 5m26s) kubelet, minikube Started container broker
Warning BackOff 28s (x15 over 5m3s) kubelet, minikube Back-off restarting failed container
As you can see, the init container starts successfully.
The broker container however seems to fail because of the readiness probe, which needs to connect to port :9092. I didn't change the initial configuration.
Am I missing something? I am pretty new to Kubernetes.
I also noticed this error message error while running "VolumeBinding" filter plugin for pod "kafka-0": pod has unbound immediate PersistentVolumeClaims, could it be related to my issue?
Thank you!
solsson commented
You have a CrashLoopBackOff which hints that the container started, i.e. the VolumeBinding error was temporary. You should check the logs from your kafka pods.
thomas-bouvier commented
Hi @solsson, thanks for your reply,
I checked these logs but didn't notice anything wrong:
kubectl -n kafka logs kafka-0
[2020-01-17 16:24:26,199] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-01-17 16:24:27,036] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2020-01-17 16:24:27,037] INFO starting (kafka.server.KafkaServer)
[2020-01-17 16:24:27,038] INFO Connecting to zookeeper on zookeeper:2181 (kafka.server.KafkaServer)
[2020-01-17 16:24:27,072] INFO [ZooKeeperClient Kafka server] Initializing a new session to zookeeper:2181. (kafka.zookeeper.ZooKeeperClient)
[2020-01-17 16:24:27,079] INFO Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,079] INFO Client environment:host.name=kafka-0.broker.kafka.svc.cluster.local (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,079] INFO Client environment:java.version=11.0.4 (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,079] INFO Client environment:java.vendor=AdoptOpenJDK (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,079] INFO Client environment:java.home=/opt/java/openjdk (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,080] INFO Client environment:java.class.path=/opt/kafka/libs/extensions/*:/opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.5.0.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/connect-api-2.3.0.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-2.3.0.jar:/opt/kafka/bin/../libs/connect-file-2.3.0.jar:/opt/kafka/bin/../libs/connect-json-2.3.0.jar:/opt/kafka/bin/../libs/connect-runtime-2.3.0.jar:/opt/kafka/bin/../libs/connect-transforms-2.3.0.jar:/opt/kafka/bin/../libs/extensions:/opt/kafka/bin/../libs/guava-20.0.jar:/opt/kafka/bin/../libs/hk2-api-2.5.0.jar:/opt/kafka/bin/../libs/hk2-locator-2.5.0.jar:/opt/kafka/bin/../libs/hk2-utils-2.5.0.jar:/opt/kafka/bin/../libs/jackson-annotations-2.9.9.jar:/opt/kafka/bin/../libs/jackson-core-2.9.9.jar:/opt/kafka/bin/../libs/jackson-databind-2.9.9.jar:/opt/kafka/bin/../libs/jackson-dataformat-csv-2.9.9.jar:/opt/kafka/bin/../libs/jackson-datatype-jdk8-2.9.9.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.9.9.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.9.9.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.9.9.jar:/opt/kafka/bin/../libs/jackson-module-paranamer-2.9.9.jar:/opt/kafka/bin/../libs/jackson-module-scala_2.12-2.9.9.jar:/opt/kafka/bin/../libs/jakarta.annotation-api-1.3.4.jar:/opt/kafka/bin/../libs/jakarta.inject-2.5.0.jar:/opt/kafka/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/opt/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.28.jar:/opt/kafka/bin/../libs/jersey-common-2.28.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.28.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.28.jar:/opt/kafka/bin/../libs/jersey-hk2-2.28.jar:/opt/kafka/bin/../libs/jersey-media-jaxb-2.28.jar:/opt/kafka/bin/../libs/jersey-server-2.28.jar:/opt/kafka/bin/../libs/jetty-client-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-continuation-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-http-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-io-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-security-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-server-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jetty-util-9.4.18.v20190429.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/jsr305-3.0.2.jar:/opt/kafka/bin/../libs/kafka_2.12-2.3.0.jar:/opt/kafka/bin/../libs/kafka_2.12-2.3.0-sources.jar:/opt/kafka/bin/../libs/kafka-clients-2.3.0.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-2.3.0.jar:/opt/kafka/bin/../libs/kafka-streams-2.3.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-2.3.0.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.12-2.3.0.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-2.3.0.jar:/opt/kafka/bin/../libs/kafka-tools-2.3.0.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-java-1.6.0.jar:/opt/kafka/bin/../libs/maven-artifact-3.6.1.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/bin/../libs/paranamer-2.8.jar:/opt/kafka/bin/../libs/plexus-utils-3.2.0.jar:/opt/kafka/bin/../libs/reflections-0.9.11.jar:/opt/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/opt/kafka/bin/../libs/scala-library-2.12.8.jar:/opt/kafka/bin/../libs/scala-logging_2.12-3.9.0.jar:/opt/kafka/bin/../libs/scala-reflect-2.12.8.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.26.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.26.jar:/opt/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/opt/kafka/bin/../libs/spotbugs-annotations-3.1.9.jar:/opt/kafka/bin/../libs/validation-api-2.0.1.Final.jar:/opt/kafka/bin/../libs/zkclient-0.11.jar:/opt/kafka/bin/../libs/zookeeper-3.4.14.jar:/opt/kafka/bin/../libs/zstd-jni-1.4.0-1.jar (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,080] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,080] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,080] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,081] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,081] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,081] INFO Client environment:os.version=4.19.81 (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,081] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,081] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,082] INFO Client environment:user.dir=/opt/kafka (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,083] INFO Initiating client connection, connectString=zookeeper:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@7776ab (org.apache.zookeeper.ZooKeeper)
[2020-01-17 16:24:27,102] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2020-01-17 16:24:27,110] INFO Opening socket connection to server zookeeper/10.96.43.109:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-01-17 16:24:27,118] INFO Socket connection established to zookeeper/10.96.43.109:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2020-01-17 16:24:27,139] INFO Session establishment complete on server zookeeper/10.96.43.109:2181, sessionid = 0x10000075c350000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2020-01-17 16:24:27,148] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2020-01-17 16:24:27,675] INFO Cluster ID = xKLWrDYIRRGSwQqdS37y_w (kafka.server.KafkaServer)
[2020-01-17 16:24:27,836] INFO KafkaConfig values:
advertised.host.name = null
advertised.listeners = PLAINTEXT://:9092,OUTSIDE://192.168.64.9:32400
advertised.port = null
alter.config.policy.class.name = null
alter.log.dirs.replication.quota.window.num = 11
alter.log.dirs.replication.quota.window.size.seconds = 1
authorizer.class.name =
auto.create.topics.enable = false
auto.leader.rebalance.enable = true
background.threads = 10
broker.id = 0
broker.id.generation.enable = true
broker.rack = null
client.quota.callback.class = null
compression.type = producer
connection.failed.authentication.delay.ms = 100
connections.max.idle.ms = 600000
connections.max.reauth.ms = 0
control.plane.listener.name = null
controlled.shutdown.enable = true
controlled.shutdown.max.retries = 3
controlled.shutdown.retry.backoff.ms = 5000
controller.socket.timeout.ms = 30000
create.topic.policy.class.name = null
default.replication.factor = 3
delegation.token.expiry.check.interval.ms = 3600000
delegation.token.expiry.time.ms = 86400000
delegation.token.master.key = null
delegation.token.max.lifetime.ms = 604800000
delete.records.purgatory.purge.interval.requests = 1
delete.topic.enable = true
fetch.purgatory.purge.interval.requests = 1000
group.initial.rebalance.delay.ms = 3000
group.max.session.timeout.ms = 1800000
group.max.size = 2147483647
group.min.session.timeout.ms = 6000
host.name =
inter.broker.listener.name = PLAINTEXT
inter.broker.protocol.version = 2.3-IV1
kafka.metrics.polling.interval.secs = 10
kafka.metrics.reporters = []
leader.imbalance.check.interval.seconds = 300
leader.imbalance.per.broker.percentage = 10
listener.security.protocol.map = PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,OUTSIDE:PLAINTEXT
listeners = PLAINTEXT://:9092,OUTSIDE://:9094
log.cleaner.backoff.ms = 15000
log.cleaner.dedupe.buffer.size = 134217728
log.cleaner.delete.retention.ms = 86400000
log.cleaner.enable = true
log.cleaner.io.buffer.load.factor = 0.9
log.cleaner.io.buffer.size = 524288
log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
log.cleaner.max.compaction.lag.ms = 9223372036854775807
log.cleaner.min.cleanable.ratio = 0.5
log.cleaner.min.compaction.lag.ms = 0
log.cleaner.threads = 1
log.cleanup.policy = [delete]
log.dir = /tmp/kafka-logs
log.dirs = /var/lib/kafka/data/topics
log.flush.interval.messages = 9223372036854775807
log.flush.interval.ms = null
log.flush.offset.checkpoint.interval.ms = 60000
log.flush.scheduler.interval.ms = 9223372036854775807
log.flush.start.offset.checkpoint.interval.ms = 60000
log.index.interval.bytes = 4096
log.index.size.max.bytes = 10485760
log.message.downconversion.enable = true
log.message.format.version = 2.3-IV1
log.message.timestamp.difference.max.ms = 9223372036854775807
log.message.timestamp.type = CreateTime
log.preallocate = false
log.retention.bytes = -1
log.retention.check.interval.ms = 300000
log.retention.hours = -1
log.retention.minutes = null
log.retention.ms = null
log.roll.hours = 168
log.roll.jitter.hours = 0
log.roll.jitter.ms = null
log.roll.ms = null
log.segment.bytes = 1073741824
log.segment.delete.delay.ms = 60000
max.connections = 2147483647
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides =
max.incremental.fetch.session.cache.slots = 1000
message.max.bytes = 1000012
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
min.insync.replicas = 2
num.io.threads = 8
num.network.threads = 3
num.partitions = 12
num.recovery.threads.per.data.dir = 1
num.replica.alter.log.dirs.threads = null
num.replica.fetchers = 1
offset.metadata.max.bytes = 4096
offsets.commit.required.acks = -1
offsets.commit.timeout.ms = 5000
offsets.load.buffer.size = 5242880
offsets.retention.check.interval.ms = 600000
offsets.retention.minutes = 10080
offsets.topic.compression.codec = 0
offsets.topic.num.partitions = 50
offsets.topic.replication.factor = 3
offsets.topic.segment.bytes = 104857600
password.encoder.cipher.algorithm = AES/CBC/PKCS5Padding
password.encoder.iterations = 4096
password.encoder.key.length = 128
password.encoder.keyfactory.algorithm = null
password.encoder.old.secret = null
password.encoder.secret = null
port = 9092
principal.builder.class = null
producer.purgatory.purge.interval.requests = 1000
queued.max.request.bytes = -1
queued.max.requests = 500
quota.consumer.default = 9223372036854775807
quota.producer.default = 9223372036854775807
quota.window.num = 11
quota.window.size.seconds = 1
replica.fetch.backoff.ms = 1000
replica.fetch.max.bytes = 1048576
replica.fetch.min.bytes = 1
replica.fetch.response.max.bytes = 10485760
replica.fetch.wait.max.ms = 500
replica.high.watermark.checkpoint.interval.ms = 5000
replica.lag.time.max.ms = 10000
replica.socket.receive.buffer.bytes = 65536
replica.socket.timeout.ms = 30000
replication.quota.window.num = 11
replication.quota.window.size.seconds = 1
request.timeout.ms = 30000
reserved.broker.max.id = 1000
sasl.client.callback.handler.class = null
sasl.enabled.mechanisms = [GSSAPI]
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.principal.to.local.rules = [DEFAULT]
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism.inter.broker.protocol = GSSAPI
sasl.server.callback.handler.class = null
security.inter.broker.protocol = PLAINTEXT
socket.receive.buffer.bytes = 102400
socket.request.max.bytes = 104857600
socket.send.buffer.bytes = 102400
ssl.cipher.suites = []
ssl.client.auth = none
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.principal.mapping.rules = [DEFAULT]
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.abort.timed.out.transaction.cleanup.interval.ms = 60000
transaction.max.timeout.ms = 900000
transaction.remove.expired.transaction.cleanup.interval.ms = 3600000
transaction.state.log.load.buffer.size = 5242880
transaction.state.log.min.isr = 2
transaction.state.log.num.partitions = 50
transaction.state.log.replication.factor = 3
transaction.state.log.segment.bytes = 104857600
transactional.id.expiration.ms = 604800000
unclean.leader.election.enable = false
zookeeper.connect = zookeeper:2181
zookeeper.connection.timeout.ms = null
zookeeper.max.in.flight.requests = 10
zookeeper.session.timeout.ms = 6000
zookeeper.set.acl = false
zookeeper.sync.time.ms = 2000
(kafka.server.KafkaConfig)
[2020-01-17 16:24:27,873] INFO KafkaConfig values:
advertised.host.name = null
advertised.listeners = PLAINTEXT://:9092,OUTSIDE://192.168.64.9:32400
advertised.port = null
alter.config.policy.class.name = null
alter.log.dirs.replication.quota.window.num = 11
alter.log.dirs.replication.quota.window.size.seconds = 1
authorizer.class.name =
auto.create.topics.enable = false
auto.leader.rebalance.enable = true
background.threads = 10
broker.id = 0
broker.id.generation.enable = true
broker.rack = null
client.quota.callback.class = null
compression.type = producer
connection.failed.authentication.delay.ms = 100
connections.max.idle.ms = 600000
connections.max.reauth.ms = 0
control.plane.listener.name = null
controlled.shutdown.enable = true
controlled.shutdown.max.retries = 3
controlled.shutdown.retry.backoff.ms = 5000
controller.socket.timeout.ms = 30000
create.topic.policy.class.name = null
default.replication.factor = 3
delegation.token.expiry.check.interval.ms = 3600000
delegation.token.expiry.time.ms = 86400000
delegation.token.master.key = null
delegation.token.max.lifetime.ms = 604800000
delete.records.purgatory.purge.interval.requests = 1
delete.topic.enable = true
fetch.purgatory.purge.interval.requests = 1000
group.initial.rebalance.delay.ms = 3000
group.max.session.timeout.ms = 1800000
group.max.size = 2147483647
group.min.session.timeout.ms = 6000
host.name =
inter.broker.listener.name = PLAINTEXT
inter.broker.protocol.version = 2.3-IV1
kafka.metrics.polling.interval.secs = 10
kafka.metrics.reporters = []
leader.imbalance.check.interval.seconds = 300
leader.imbalance.per.broker.percentage = 10
listener.security.protocol.map = PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,OUTSIDE:PLAINTEXT
listeners = PLAINTEXT://:9092,OUTSIDE://:9094
log.cleaner.backoff.ms = 15000
log.cleaner.dedupe.buffer.size = 134217728
log.cleaner.delete.retention.ms = 86400000
log.cleaner.enable = true
log.cleaner.io.buffer.load.factor = 0.9
log.cleaner.io.buffer.size = 524288
log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
log.cleaner.max.compaction.lag.ms = 9223372036854775807
log.cleaner.min.cleanable.ratio = 0.5
log.cleaner.min.compaction.lag.ms = 0
log.cleaner.threads = 1
log.cleanup.policy = [delete]
log.dir = /tmp/kafka-logs
log.dirs = /var/lib/kafka/data/topics
log.flush.interval.messages = 9223372036854775807
log.flush.interval.ms = null
log.flush.offset.checkpoint.interval.ms = 60000
log.flush.scheduler.interval.ms = 9223372036854775807
log.flush.start.offset.checkpoint.interval.ms = 60000
log.index.interval.bytes = 4096
log.index.size.max.bytes = 10485760
log.message.downconversion.enable = true
log.message.format.version = 2.3-IV1
log.message.timestamp.difference.max.ms = 9223372036854775807
log.message.timestamp.type = CreateTime
log.preallocate = false
log.retention.bytes = -1
log.retention.check.interval.ms = 300000
log.retention.hours = -1
log.retention.minutes = null
log.retention.ms = null
log.roll.hours = 168
log.roll.jitter.hours = 0
log.roll.jitter.ms = null
log.roll.ms = null
log.segment.bytes = 1073741824
log.segment.delete.delay.ms = 60000
max.connections = 2147483647
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides =
max.incremental.fetch.session.cache.slots = 1000
message.max.bytes = 1000012
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
min.insync.replicas = 2
num.io.threads = 8
num.network.threads = 3
num.partitions = 12
num.recovery.threads.per.data.dir = 1
num.replica.alter.log.dirs.threads = null
num.replica.fetchers = 1
offset.metadata.max.bytes = 4096
offsets.commit.required.acks = -1
offsets.commit.timeout.ms = 5000
offsets.load.buffer.size = 5242880
offsets.retention.check.interval.ms = 600000
offsets.retention.minutes = 10080
offsets.topic.compression.codec = 0
offsets.topic.num.partitions = 50
offsets.topic.replication.factor = 3
offsets.topic.segment.bytes = 104857600
password.encoder.cipher.algorithm = AES/CBC/PKCS5Padding
password.encoder.iterations = 4096
password.encoder.key.length = 128
password.encoder.keyfactory.algorithm = null
password.encoder.old.secret = null
password.encoder.secret = null
port = 9092
principal.builder.class = null
producer.purgatory.purge.interval.requests = 1000
queued.max.request.bytes = -1
queued.max.requests = 500
quota.consumer.default = 9223372036854775807
quota.producer.default = 9223372036854775807
quota.window.num = 11
quota.window.size.seconds = 1
replica.fetch.backoff.ms = 1000
replica.fetch.max.bytes = 1048576
replica.fetch.min.bytes = 1
replica.fetch.response.max.bytes = 10485760
replica.fetch.wait.max.ms = 500
replica.high.watermark.checkpoint.interval.ms = 5000
replica.lag.time.max.ms = 10000
replica.socket.receive.buffer.bytes = 65536
replica.socket.timeout.ms = 30000
replication.quota.window.num = 11
replication.quota.window.size.seconds = 1
request.timeout.ms = 30000
reserved.broker.max.id = 1000
sasl.client.callback.handler.class = null
sasl.enabled.mechanisms = [GSSAPI]
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.principal.to.local.rules = [DEFAULT]
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism.inter.broker.protocol = GSSAPI
sasl.server.callback.handler.class = null
security.inter.broker.protocol = PLAINTEXT
socket.receive.buffer.bytes = 102400
socket.request.max.bytes = 104857600
socket.send.buffer.bytes = 102400
ssl.cipher.suites = []
ssl.client.auth = none
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.principal.mapping.rules = [DEFAULT]
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.abort.timed.out.transaction.cleanup.interval.ms = 60000
transaction.max.timeout.ms = 900000
transaction.remove.expired.transaction.cleanup.interval.ms = 3600000
transaction.state.log.load.buffer.size = 5242880
transaction.state.log.min.isr = 2
transaction.state.log.num.partitions = 50
transaction.state.log.replication.factor = 3
transaction.state.log.segment.bytes = 104857600
transactional.id.expiration.ms = 604800000
unclean.leader.election.enable = false
zookeeper.connect = zookeeper:2181
zookeeper.connection.timeout.ms = null
zookeeper.max.in.flight.requests = 10
zookeeper.session.timeout.ms = 6000
zookeeper.set.acl = false
zookeeper.sync.time.ms = 2000
(kafka.server.KafkaConfig)
[2020-01-17 16:24:27,988] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2020-01-17 16:24:27,995] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2020-01-17 16:24:27,995] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2020-01-17 16:24:28,065] INFO Loading logs. (kafka.log.LogManager)
[2020-01-17 16:24:28,084] INFO Logs loading complete in 19 ms. (kafka.log.LogManager)
[2020-01-17 16:24:28,108] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2020-01-17 16:24:28,115] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2020-01-17 16:24:29,033] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2020-01-17 16:24:29,107] INFO [SocketServer brokerId=0] Created data-plane acceptor and processors for endpoint : EndPoint(null,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)
[2020-01-17 16:24:29,107] INFO Awaiting socket connections on 0.0.0.0:9094. (kafka.network.Acceptor)
[2020-01-17 16:24:29,137] INFO [SocketServer brokerId=0] Created data-plane acceptor and processors for endpoint : EndPoint(null,9094,ListenerName(OUTSIDE),PLAINTEXT) (kafka.network.SocketServer)
[2020-01-17 16:24:29,139] INFO [SocketServer brokerId=0] Started 2 acceptor threads for data-plane (kafka.network.SocketServer)
[2020-01-17 16:24:29,270] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2020-01-17 16:24:29,277] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2020-01-17 16:24:29,277] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2020-01-17 16:24:29,286] INFO [ExpirationReaper-0-ElectPreferredLeader]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2020-01-17 16:24:29,344] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler)
[2020-01-17 16:24:29,432] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.zk.KafkaZkClient)
[2020-01-17 16:24:29,467] INFO Stat of the created znode at /brokers/ids/0 is: 12884901977,12884901977,1579278269446,1579278269446,1,0,0,72057625649676288,301,0,12884901977
(kafka.zk.KafkaZkClient)
[2020-01-17 16:24:29,468] INFO Registered broker 0 at path /brokers/ids/0 with addresses: ArrayBuffer(EndPoint(kafka-0.broker.kafka.svc.cluster.local,9092,ListenerName(PLAINTEXT),PLAINTEXT), EndPoint(192.168.64.9,32400,ListenerName(OUTSIDE),PLAINTEXT)), czxid (broker epoch): 12884901977 (kafka.zk.KafkaZkClient)
[2020-01-17 16:24:29,552] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2020-01-17 16:24:29,587] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2020-01-17 16:24:29,599] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2020-01-17 16:24:29,634] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
[2020-01-17 16:24:29,639] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
[2020-01-17 16:24:29,670] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 29 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2020-01-17 16:24:29,756] INFO [ProducerId Manager 0]: Acquired new producerId block (brokerId:0,blockStartProducerId:5000,blockEndProducerId:5999) by writing to Zk with path version 6 (kafka.coordinator.transaction.ProducerIdManager)
[2020-01-17 16:24:29,823] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2020-01-17 16:24:29,825] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2020-01-17 16:24:29,854] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2020-01-17 16:24:30,041] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2020-01-17 16:24:30,414] INFO [SocketServer brokerId=0] Started data-plane processors for 2 acceptors (kafka.network.SocketServer)
[2020-01-17 16:24:30,497] INFO Kafka version: 2.3.0 (org.apache.kafka.common.utils.AppInfoParser)
[2020-01-17 16:24:30,497] INFO Kafka commitId: fc1aaa116b661c8a (org.apache.kafka.common.utils.AppInfoParser)
[2020-01-17 16:24:30,498] INFO Kafka startTimeMs: 1579278270415 (org.apache.kafka.common.utils.AppInfoParser)
[2020-01-17 16:24:30,514] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
kubectl -n kafka logs kafka-0 -c init-config
+ cp /etc/kafka-configmap/log4j.properties /etc/kafka/
+ KAFKA_BROKER_ID=0
+ SEDS=("s/#init#broker.id=#init#/broker.id=$KAFKA_BROKER_ID/")
+ LABELS=kafka-broker-id=0
+ ANNOTATIONS=
+ hash kubectl
++ kubectl get node minikube '-o=go-template={{index .metadata.labels "failure-domain.beta.kubernetes.io/zone"}}'
+ ZONE='<no value>'
+ '[' 'x<no value>' == 'x<no value>' ']'
+ SEDS+=("s/#init#broker.rack=#init#/#init#broker.rack=# zone label not found for node $NODE_NAME/")
++ kubectl get node minikube -o 'jsonpath={.status.addresses[?(@.type=="InternalIP")].address}'
+ OUTSIDE_HOST=192.168.64.9
+ OUTSIDE_PORT=32400
+ SEDS+=("s|#init#advertised.listeners=PLAINTEXT://#init#|advertised.listeners=PLAINTEXT://:9092,OUTSIDE://${OUTSIDE_HOST}:${OUTSIDE_PORT}|")
+ ANNOTATIONS=' kafka-listener-outside-host=192.168.64.9 kafka-listener-outside-port=32400'
+ '[' '!' -z kafka-broker-id=0 ']'
+ kubectl -n kafka label pod kafka-0 kafka-broker-id=0
pod/kafka-0 labeled
+ '[' '!' -z ' kafka-listener-outside-host=192.168.64.9 kafka-listener-outside-port=32400' ']'
+ kubectl -n kafka annotate pod kafka-0 kafka-listener-outside-host=192.168.64.9 kafka-listener-outside-port=32400
pod/kafka-0 annotated
+ printf '%s\n' s/#init#broker.id=#init#/broker.id=0/ 's/#init#broker.rack=#init#/#init#broker.rack=# zone label not found for node minikube/' 's|#init#advertised.listeners=PLAINTEXT://#init#|advertised.listeners=PLAINTEXT://:9092,OUTSIDE://192.168.64.9:32400|'
+ sed -f - /etc/kafka-configmap/server.properties
+ '[' 0 -eq 0 ']'
+ mv /etc/kafka/server.properties.tmp /etc/kafka/server.properties
The output is similar for kafka-1 and kafka-2.
solsson commented
I think it's quite local to your environment. I'd try to remove the liveness probe and experiment from there, with the containers running.
Ritwikkumarr commented
Hi Solsson
any update on this readiness probe failure because i am trying to deploy kafka on AKS, zookeeper runs fine its only kafka broker pods that fail due to readiness probe failure. Please can you update us on this matter
AkshayAwate commented
I have tried on ec2 as nodes, I am facing same issue
running "VolumeBinding" filter plugin for pod "kafka-0": pod has unbound immediate PersistentVolumeClaims