PVC stuck in pending state
Opened this issue · 0 comments
below are the logs of rbd provisioner
kubectl logs -f -n ceph ceph-rbd-provisioner-5b9bfb859d-2wvd9
- exec /usr/local/bin/rbd-provisioner -id ceph-rbd-provisioner-5b9bfb859d-2wvd9
I0514 08:59:52.174721 1 main.go:84] Creating RBD provisioner with identity: ceph-rbd-provisioner-5b9bfb859d-2wvd9
I0514 08:59:52.225339 1 controller.go:407] Starting provisioner controller a2b23724-7626-11e9-9f90-0242ac1e4803!
I0514 09:13:36.115718 1 controller.go:1068] scheduleOperation[lock-provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:13:36.222196 1 controller.go:1068] scheduleOperation[lock-provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:13:36.300492 1 leaderelection.go:156] attempting to acquire leader lease...
E0514 09:13:36.371424 1 leaderelection.go:273] Failed to update lock: Operation cannot be fulfilled on persistentvolumeclaims "ceph-pvc": the object has been modified; please apply your changes to the latest version and try again
I0514 09:13:37.253079 1 controller.go:1068] scheduleOperation[lock-provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:13:38.408582 1 controller.go:1068] scheduleOperation[lock-provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:13:52.253423 1 controller.go:1068] scheduleOperation[lock-provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:14:07.254063 1 controller.go:1068] scheduleOperation[lock-provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:14:22.254484 1 controller.go:1068] scheduleOperation[lock-provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:14:24.849756 1 leaderelection.go:178] successfully acquired lease to provision for pvc default/ceph-pvc
I0514 09:14:24.850364 1 controller.go:1068] scheduleOperation[provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:14:37.254732 1 controller.go:1068] scheduleOperation[provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:14:52.255130 1 controller.go:1068] scheduleOperation[provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:14:55.219073 1 leaderelection.go:204] stopped trying to renew lease to provision for pvc default/ceph-pvc, timeout reached
I0514 09:15:07.255355 1 controller.go:1068] scheduleOperation[provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:15:22.255579 1 controller.go:1068] scheduleOperation[lock-provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:15:22.267831 1 leaderelection.go:156] attempting to acquire leader lease...
I0514 09:15:37.256247 1 controller.go:1068] scheduleOperation[lock-provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:15:52.256693 1 controller.go:1068] scheduleOperation[lock-provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:16:04.686503 1 leaderelection.go:178] successfully acquired lease to provision for pvc default/ceph-pvc
I0514 09:16:04.686626 1 controller.go:1068] scheduleOperation[provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:16:07.256853 1 controller.go:1068] scheduleOperation[provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:16:22.257263 1 controller.go:1068] scheduleOperation[provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:16:35.201268 1 leaderelection.go:204] stopped trying to renew lease to provision for pvc default/ceph-pvc, timeout reached
I0514 09:16:37.257458 1 controller.go:1068] scheduleOperation[provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:16:52.258008 1 controller.go:1068] scheduleOperation[lock-provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:16:52.275487 1 leaderelection.go:156] attempting to acquire leader lease...
I0514 09:16:52.302071 1 leaderelection.go:178] successfully acquired lease to provision for pvc default/ceph-pvc
I0514 09:16:52.302256 1 controller.go:1068] scheduleOperation[provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:17:07.258228 1 controller.go:1068] scheduleOperation[provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:17:22.258531 1 controller.go:1068] scheduleOperation[provision-default/ceph-pvc[8dc8b0cc-7628-11e9-bef5-080027176a7d]]
I0514 09:17:22.658162 1 leaderelection.go:204] stopped trying to renew lease to provision for pvc default/ceph-pvc, timeout reached
All pods are running well
kubectl get pods -n ceph
NAME READY STATUS RESTARTS AGE
ceph-mds-68c79b5cc-jt2m9 0/1 Pending 0 47m
ceph-mds-keyring-generator-jmj5g 0/1 Completed 0 47m
ceph-mgr-6c687f5964-7g8fq 1/1 Running 1 47m
ceph-mgr-keyring-generator-4jtbm 0/1 Completed 0 47m
ceph-mon-check-676d984874-mnscz 1/1 Running 0 47m
ceph-mon-fl2jl 3/3 Running 0 48m
ceph-mon-keyring-generator-cv8ff 0/1 Completed 0 47m
ceph-namespace-client-key-generator-klnmt 0/1 Completed 0 47m
ceph-osd-dev-sdb-hnkd4 1/1 Running 0 47m
ceph-osd-keyring-generator-7h7dz 0/1 Completed 0 47m
ceph-rbd-provisioner-5b9bfb859d-2wvd9 1/1 Running 0 47m
ceph-rbd-provisioner-5b9bfb859d-r7khc 1/1 Running 0 47m
ceph-rgw-6d946b-ztwnx 0/1 Pending 0 47m
ceph-rgw-keyring-generator-ndzbz 0/1 Completed 0 47m
ceph-storage-keys-generator-2wnrq 0/1 Completed 0 47m
3 nodes one node for osd one for mon one master node for kubernetes