Couldn't add persistent volume claims

Question

Couldn't add persistent volume claims

Closed this issue 7 years ago · 17 comments

The headless worked great. But when trying the stateful service, the persistent volumes were created successfully, but the persistent volume claim keeps showing pending, and never bound successfully.

Answer 1 · 2017-10-09T18:34:11.000Z

What does it show when you execute this command: kubectl get pv,pvc

Answer 2 · 2017-10-09T18:35:34.000Z

'get pv' shows all the 3 volumes created successfully, but 'get pvc' shows the claim is pending (after a long time"

Answer 3 · 2017-10-09T18:38:49.000Z

here is a screenshot:

Answer 4 · 2017-10-09T18:44:47.000Z

It looks like you are using a paid cluster on Bluemix Container service. The pvc will be provisioned dynamically which means you wouldn't need to create Persistent Volumes. The storage class (ibmc-file-bronze) will create that for you whenever a persistent volume claim is created. It should get provisioned in less than 5 minutes

Answer 5 · 2017-10-09T18:46:34.000Z

I see. Should I just delete all the local volumes, and then rerun to create the statefulset? Thanks.

Answer 6 · 2017-10-09T18:56:26.000Z

Just deleted all the local volumes, statefulset, and then rerun to create the statefulset. Didn't see any PV created automatically. The PVC still shows pending.

Answer 7 · 2017-10-09T19:00:32.000Z

Can you do bx cs cluster-get YOUR_CLUSTER_NAME

It should show the addons in your cluster:

...
Addons
Name                   Enabled   
customer-storage-pod   true   
basic-ingress          true   
storage-watcher-pod    true

Answer 8 · 2017-10-09T19:03:49.000Z

It doesn't show the addon. How can I install it? Here is my screenshot:

Answer 9 · 2017-10-09T20:21:26.000Z

Try updating the bx cli bx plugin update container-service -r Bluemix

Answer 10 · 2017-10-09T21:48:00.000Z

Thanks. After updating the addon, I now do see the following. But I have tried to redeploy everything and still not working

Answer 11 · 2017-10-09T22:08:06.000Z

Try doing kubectl get pods -n kube-system and let me know if there is a pod named ibm-storage-watcher-[xxxxx]

Answer 12 · 2017-10-09T22:11:07.000Z

Yes, I do have a pod ibm-storage-watcher-2152805327-ccln6 running. Also every time, I run the statefulset creation, I did get a notification that a new storage order was approved, but not sure why I couldn't see the actual storage.

Answer 13 · 2017-10-09T22:36:43.000Z

The problem might be on the Softlayer side. You cloud try manually creating a Persistent Volume that uses that file storage in softlayer. https://console.bluemix.net/docs/containers/cs_cluster.html#cs_cluster_volume_create

--- or ---

You can specify to not use the storage class (by setting annotation of storage class to empty) and use the local-volumes.yaml
You could modify the cassandra-statefulset.yaml

...
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
      annotations:
        volume.beta.kubernetes.io/storage-class: ""
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

That should use the volumes created through local-volumes.yaml (It's not gonna be using the file storage in Softlayer)

Answer 14 · 2017-10-09T22:59:36.000Z

Thanks. It did work. But when I scaled it to 4 replica sets, the other 3 crashed, not sure why. See screenshot below:

Answer 15 · 2017-10-09T23:00:28.000Z

Here is the screenshot of the bounded pvc requests:

Answer 16 · 2017-10-10T16:25:00.000Z

Any logs on the cassandra-1,2,3? kubectl logs cassandra-1. Also check for the cassandra service kubectl get svc

Answer 17 · 2017-10-10T16:59:18.000Z

Thanks. The log files did show the error from the cassandra seed variable. If I comment it out, it works fine. Not sure if there is a better way to do it. Thanks.