Yolean/kubernetes-kafka

Unable to set up PVC with kafka and zookeeper statefulsets

ishantanu opened this issue · 5 comments

ishantanu commented

Hi,
I am trying to set up Kafka on k8s cluster on AWS. While I was following the instructions as mentioned here, I faced an issue where my PVC's were not able to bound with the pods. Here's are some details:

  1. PVC's in kafka namespace:
NAME           STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
data-kafka-0   Bound     pvc-a5000f90-be36-11e8-8790-02d3f7512070   500Gi      RWO            kafka-broker      11h
data-pzoo-0    Bound     pvc-366fb773-be35-11e8-8790-02d3f7512070   1Gi        RWO            kafka-zookeeper   11h
  1. Statefulsets:
NAME      DESIRED   CURRENT   AGE
kafka     3         1         11h
pzoo      3         1         12h
zoo       2         2         12h
  1. Pods in kafka ns:
NAME      READY     STATUS     RESTARTS   AGE
kafka-0   0/1       Init:0/1   0          11h
pzoo-0    0/1       Init:0/1   0          11h
zoo-0     1/1       Running    0          11h
zoo-1     1/1       Running    0          11h
  1. Describing kafka-0 pod issue:
  Warning  FailedMount  10m (x309 over 11h)  kubelet, ip-127-80-91-9.us-east-2.compute.internal  Unable to mount volumes for pod "kafka-0_kafka(dd0197cd-be35-11e8-8790-02d3f7512070)": timeout expired waiting for volumes to attach or mount for pod "kafka"/"kafka-0". list of unmounted volumes=[data]. list of unattached volumes=[data configmap config default-token-56j8c]
  Warning  FailedMount  4m (x60 over 11h)    kubelet, ip-172-20-41-9.us-east-2.compute.internal  MountVolume.WaitForAttach failed for volume "pvc-a5000f90-be36-11e8-8790-02d3f7512070" : Could not find attached AWS Volume "aws://us-east-2a/vol-07b00075a74d5ae28". Timeout waiting for mount paths to be created.

Is there anything else which needs to be done apart from the mentioned instructions?

My requirement: I need to setup a kafka on k8s having 2 brokers and having a persistent storage with ebs.

Let me know if I am missing something.

solsson commented

Which storageclasses did you use? There's two sets for AWS in ./configure. Or did you use your own?

Hermain commented

I had similar problems. These occured when I used a newer (not kops default) amazon debian image in order to be able to use m5.large instances with volumes.
If you also switched the linux version try the kops default. That should work.

ishantanu commented

@Hermain That was the issue. I was using newer m5 instances with kops. The fix was to change instance groups:

from:
image: kope.io/k8s-1.9-debian-jessie-amd64-hvm-ebs-2018-03-11
to:
image: kope.io/k8s-1.9-debian-stretch-amd64-hvm-ebs-2018-03-11

@solsson I modified storageclasses. Used st1 for storing broker data and gp2 for zookeeper and all is working okay. I just had a rookie question: After applying outside-services for exposing brokers with NodePort, how I can test the access to them?

solsson commented

Thanks for following up.

Regarding the OT question i think that is cluster provider specific. There are some AWS examples in the issues/PRs on external access if I'm not mistaken. The basic idea is that you point any kafka client (I'm a big fan of kafkacat with -d broker for this kind of test) to any of the nodes' port 32400 and let the bootstrap process do its work. That shows if you got the listener name resolution right in the init script.

ishantanu commented

@solsson Thanks. Will give it a try.