[secrets "ceph-bootstrap-osd-keyring" not found] ceph-osd fail on k8s 1.10.1 install by kubeadm
foxundermoon opened this issue · 33 comments
foxundermoon commented
kubectl get po -n ceph
NAME READY STATUS RESTARTS AGE
ceph-mds-696bd98bdb-bnvpg 0/1 Pending 0 18m
ceph-mds-keyring-generator-q679r 0/1 Completed 0 18m
ceph-mgr-6d5f86d9c4-nr76h 1/1 Running 1 18m
ceph-mgr-keyring-generator-v825z 0/1 Completed 0 18m
ceph-mon-86lth 1/1 Running 0 18m
ceph-mon-check-74d98c5b95-wf9tm 1/1 Running 0 18m
ceph-mon-keyring-generator-rfg8j 0/1 Completed 0 18m
ceph-mon-pp5hc 1/1 Running 0 18m
ceph-namespace-client-key-cleaner-g9dri-sjmqd 0/1 Completed 0 1h
ceph-namespace-client-key-cleaner-qwkee-pdkh6 0/1 Completed 0 21m
ceph-namespace-client-key-cleaner-t25ui-5gkb7 0/1 Completed 0 2d
ceph-namespace-client-key-generator-xk4w6 0/1 Completed 0 18m
ceph-osd-dev-sda-6jbgd 0/1 Init:CrashLoopBackOff 8 18m
ceph-osd-dev-sda-khfhw 0/1 Init:CrashLoopBackOff 8 18m
ceph-osd-dev-sda-krkjf 0/1 Init:CrashLoopBackOff 8 18m
ceph-osd-keyring-generator-mvktj 0/1 Completed 0 18m
ceph-rbd-provisioner-b58659dc9-nhx2q 1/1 Running 0 18m
ceph-rbd-provisioner-b58659dc9-nnlh2 1/1 Running 0 18m
ceph-rgw-5bd9dd66c5-gh946 0/1 Pending 0 18m
ceph-rgw-keyring-generator-dz9kd 0/1 Completed 0 18m
ceph-storage-admin-key-cleaner-1as0t-fq589 0/1 Completed 0 1h
ceph-storage-admin-key-cleaner-oayjp-fglzr 0/1 Completed 0 2d
ceph-storage-admin-key-cleaner-zemvx-jxn7c 0/1 Completed 0 21m
ceph-storage-keys-generator-szps9 0/1 Completed 0 18m
Version of Helm and Kubernetes:
- k8s 1.10.1
- helm
helm version
Client: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Which chart:
commit 70681c8218e75bb10acc7dc210b791b69545ce0d
Merge: a4fa8a1 6cb2e1e
Author: Huamin Chen <rootfs@users.noreply.github.com>
Date: Thu Apr 26 12:11:43 2018 -0400
- pod describe
kubectl describe -n ceph po ceph-osd-dev-sda-6jbgd
Name: ceph-osd-dev-sda-6jbgd
Namespace: ceph
Node: k8s-2/192.168.16.40
Start Time: Sat, 28 Apr 2018 15:54:08 +0800
Labels: application=ceph
component=osd
controller-revision-hash=1450926272
pod-template-generation=1
release_group=ceph
Annotations: <none>
Status: Pending
IP: 192.168.16.40
Controlled By: DaemonSet/ceph-osd-dev-sda
Init Containers:
init:
Container ID: docker://8c9536cc4c5d811f57ba6349c87245121651841f52db682f858ae0ac70555856
Image: docker.io/kolla/ubuntu-source-kubernetes-entrypoint:4.0.0
Image ID: docker-pullable://kolla/ubuntu-source-kubernetes-entrypoint@sha256:75116ab2f9f65c5fc078e68ce7facd66c1c57496947f37b7209b32f94925e53b
Port: <none>
Host Port: <none>
Command:
kubernetes-entrypoint
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 28 Apr 2018 15:54:34 +0800
Finished: Sat, 28 Apr 2018 15:54:36 +0800
Ready: True
Restart Count: 0
Environment:
POD_NAME: ceph-osd-dev-sda-6jbgd (v1:metadata.name)
NAMESPACE: ceph (v1:metadata.namespace)
INTERFACE_NAME: eth0
DEPENDENCY_SERVICE: ceph-mon
DEPENDENCY_JOBS:
DEPENDENCY_DAEMONSET:
DEPENDENCY_CONTAINER:
COMMAND: echo done
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-z5m75 (ro)
ceph-init-dirs:
Container ID: docker://1562879ebbc52c47cfd9fb292339e548d26450207846ff6eeb38594569d5ec5f
Image: docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04
Image ID: docker-pullable://ceph/daemon@sha256:687056228e899ecbfd311854e3864db0b46dd4a9a6d4eb4b47c815ca413f25ee
Port: <none>
Host Port: <none>
Command:
/tmp/init_dirs.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 28 Apr 2018 15:54:38 +0800
Finished: Sat, 28 Apr 2018 15:54:39 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/run from pod-run (rw)
/tmp/init_dirs.sh from ceph-bin (ro)
/var/lib/ceph from pod-var-lib-ceph (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-z5m75 (ro)
/variables_entrypoint.sh from ceph-bin (ro)
osd-prepare-pod:
Container ID: docker://2b5bed33de8f35533eb72ef3208010153b904a8ed34c527a4916b88f549d5f6b
Image: docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04
Image ID: docker-pullable://ceph/daemon@sha256:687056228e899ecbfd311854e3864db0b46dd4a9a6d4eb4b47c815ca413f25ee
Port: <none>
Host Port: <none>
Command:
/start_osd.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 28 Apr 2018 16:11:06 +0800
Finished: Sat, 28 Apr 2018 16:11:07 +0800
Ready: False
Restart Count: 8
Environment:
CEPH_DAEMON: osd_ceph_disk_prepare
KV_TYPE: k8s
CLUSTER: ceph
CEPH_GET_ADMIN_KEY: 1
OSD_DEVICE: /dev/mapper/centos-root
HOSTNAME: (v1:spec.nodeName)
Mounts:
/common_functions.sh from ceph-bin (ro)
/dev from devices (rw)
/etc/ceph/ceph.client.admin.keyring from ceph-client-admin-keyring (ro)
/etc/ceph/ceph.conf from ceph-etc (ro)
/etc/ceph/ceph.mon.keyring from ceph-mon-keyring (ro)
/osd_activate_journal.sh from ceph-bin (ro)
/osd_disk_activate.sh from ceph-bin (ro)
/osd_disk_prepare.sh from ceph-bin (ro)
/osd_disks.sh from ceph-bin (ro)
/run from pod-run (rw)
/start_osd.sh from ceph-bin (ro)
/var/lib/ceph from pod-var-lib-ceph (rw)
/var/lib/ceph/bootstrap-mds/ceph.keyring from ceph-bootstrap-mds-keyring (ro)
/var/lib/ceph/bootstrap-osd/ceph.keyring from ceph-bootstrap-osd-keyring (ro)
/var/lib/ceph/bootstrap-rgw/ceph.keyring from ceph-bootstrap-rgw-keyring (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-z5m75 (ro)
/variables_entrypoint.sh from ceph-bin (ro)
Containers:
osd-activate-pod:
Container ID:
Image: docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04
Image ID:
Port: <none>
Host Port: <none>
Command:
/start_osd.sh
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Liveness: tcp-socket :6800 delay=60s timeout=5s period=10s #success=1 #failure=3
Readiness: tcp-socket :6800 delay=0s timeout=5s period=10s #success=1 #failure=3
Environment:
CEPH_DAEMON: osd_ceph_disk_activate
KV_TYPE: k8s
CLUSTER: ceph
CEPH_GET_ADMIN_KEY: 1
OSD_DEVICE: /dev/mapper/centos-root
HOSTNAME: (v1:spec.nodeName)
Mounts:
/common_functions.sh from ceph-bin (ro)
/dev from devices (rw)
/etc/ceph/ceph.client.admin.keyring from ceph-client-admin-keyring (ro)
/etc/ceph/ceph.conf from ceph-etc (ro)
/etc/ceph/ceph.mon.keyring from ceph-mon-keyring (ro)
/osd_activate_journal.sh from ceph-bin (ro)
/osd_disk_activate.sh from ceph-bin (ro)
/osd_disk_prepare.sh from ceph-bin (ro)
/osd_disks.sh from ceph-bin (ro)
/run from pod-run (rw)
/start_osd.sh from ceph-bin (ro)
/var/lib/ceph from pod-var-lib-ceph (rw)
/var/lib/ceph/bootstrap-mds/ceph.keyring from ceph-bootstrap-mds-keyring (ro)
/var/lib/ceph/bootstrap-osd/ceph.keyring from ceph-bootstrap-osd-keyring (ro)
/var/lib/ceph/bootstrap-rgw/ceph.keyring from ceph-bootstrap-rgw-keyring (ro)
/var/log/ceph from pod-var-log-ceph (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-z5m75 (ro)
/variables_entrypoint.sh from ceph-bin (ro)
Conditions:
Type Status
Initialized False
Ready False
PodScheduled True
Volumes:
devices:
Type: HostPath (bare host directory volume)
Path: /dev
HostPathType:
pod-var-lib-ceph:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
pod-var-log-ceph:
Type: HostPath (bare host directory volume)
Path: /var/log/ceph/osd
HostPathType:
pod-run:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
ceph-bin:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: ceph-bin
Optional: false
ceph-etc:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: ceph-etc
Optional: false
ceph-client-admin-keyring:
Type: Secret (a volume populated by a Secret)
SecretName: ceph-client-admin-keyring
Optional: false
ceph-mon-keyring:
Type: Secret (a volume populated by a Secret)
SecretName: ceph-mon-keyring
Optional: false
ceph-bootstrap-osd-keyring:
Type: Secret (a volume populated by a Secret)
SecretName: ceph-bootstrap-osd-keyring
Optional: false
ceph-bootstrap-mds-keyring:
Type: Secret (a volume populated by a Secret)
SecretName: ceph-bootstrap-mds-keyring
Optional: false
ceph-bootstrap-rgw-keyring:
Type: Secret (a volume populated by a Secret)
SecretName: ceph-bootstrap-rgw-keyring
Optional: false
default-token-z5m75:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-z5m75
Optional: false
QoS Class: BestEffort
Node-Selectors: ceph-osd=enabled
ceph-osd-device-dev-sda=enabled
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "pod-var-log-ceph"
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "devices"
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "pod-var-lib-ceph"
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "pod-run"
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "ceph-bin"
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "ceph-etc"
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "default-token-z5m75"
Warning FailedMount 19m (x2 over 19m) kubelet, k8s-2 MountVolume.SetUp failed for volume "ceph-bootstrap-osd-keyring" : secrets "ceph-bootstrap-osd-keyring" not found
Warning FailedMount 19m (x2 over 19m) kubelet, k8s-2 MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found
Warning FailedMount 19m (x2 over 19m) kubelet, k8s-2 MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secrets "ceph-client-admin-keyring" not found
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "ceph-client-admin-keyring"
Normal SuccessfulMountVolume 19m kubelet, k8s-2 MountVolume.SetUp succeeded for volume "ceph-bootstrap-osd-keyring"
Warning FailedMount 19m (x4 over 19m) kubelet, k8s-2 MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found
Warning FailedMount 19m (x4 over 19m) kubelet, k8s-2 MountVolume.SetUp failed for volume "ceph-bootstrap-mds-keyring" : secrets "ceph-bootstrap-mds-keyring" not found
Normal SuccessfulMountVolume 19m (x2 over 19m) kubelet, k8s-2 (combined from similar events): MountVolume.SetUp succeeded for volume "ceph-mon-keyring"
Warning BackOff 4m (x65 over 18m) kubelet, k8s-2 Back-off restarting failed container
- pod log
kubectl logs -n ceph ceph-osd-dev-sda-6jbgd
Error from server (BadRequest): container "osd-activate-pod" in pod "ceph-osd-dev-sda-6jbgd" is waiting to start: PodInitializing
foxundermoon commented
- tiller log
[tiller] 2018/04/28 07:34:20 preparing install for ceph
[storage] 2018/04/28 07:34:20 getting release history for "ceph"
[tiller] 2018/04/28 07:34:20 rendering ceph chart using values
2018/04/28 07:34:20 info: manifest "ceph/templates/configmap-bin-ks.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-ks-service.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/secret-keystone.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-bootstrap.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/daemonset-osd.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-ks-endpoints.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-ceph-commands.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-ks-user.yaml" is empty. Skipping.
[tiller] 2018/04/28 07:34:20 performing install for ceph
[tiller] 2018/04/28 07:34:20 executing 2 pre-install hooks for ceph
[tiller] 2018/04/28 07:34:20 hooks complete for pre-install ceph
[storage] 2018/04/28 07:34:20 getting release history for "ceph"
[storage] 2018/04/28 07:34:20 creating release "ceph.v1"
[kube] 2018/04/28 07:34:20 building resources from manifest
[kube] 2018/04/28 07:34:20 creating 22 resource(s)
[tiller] 2018/04/28 07:34:20 executing 2 post-install hooks for ceph
[tiller] 2018/04/28 07:34:20 hooks complete for post-install ceph
[storage] 2018/04/28 07:34:20 updating release "ceph.v1"
[storage] 2018/04/28 07:34:21 getting last revision of "ceph"
[storage] 2018/04/28 07:34:21 getting release history for "ceph"
[kube] 2018/04/28 07:34:21 Doing get for Secret: "ceph-keystone-user-rgw"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Secret/ceph-keystone-user-rgw
[kube] 2018/04/28 07:34:21 Doing get for ConfigMap: "ceph-bin-clients"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/ConfigMap/ceph-bin-clients
[kube] 2018/04/28 07:34:21 Doing get for ConfigMap: "ceph-bin"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/ConfigMap/ceph-bin
[kube] 2018/04/28 07:34:21 Doing get for ConfigMap: "ceph-etc"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/ConfigMap/ceph-etc
[kube] 2018/04/28 07:34:21 Doing get for ConfigMap: "ceph-templates"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/ConfigMap/ceph-templates
[kube] 2018/04/28 07:34:21 Doing get for StorageClass: "ceph-rbd"
[kube] 2018/04/28 07:34:21 get relation pod of object: /StorageClass/ceph-rbd
[kube] 2018/04/28 07:34:21 Doing get for Service: "ceph-mon"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Service/ceph-mon
[kube] 2018/04/28 07:34:21 Doing get for Service: "ceph-rgw"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Service/ceph-rgw
[kube] 2018/04/28 07:34:21 Doing get for DaemonSet: "ceph-mon"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/DaemonSet/ceph-mon
[kube] 2018/04/28 07:34:21 Doing get for DaemonSet: "ceph-osd-dev-sda"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/DaemonSet/ceph-osd-dev-sda
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-mds"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-mds
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-mgr"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-mgr
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-mon-check"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-mon-check
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-rbd-provisioner"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-rbd-provisioner
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-rgw"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-rgw
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-mds-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-mds-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-mgr-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-mgr-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-mon-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-mon-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-rgw-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-rgw-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-osd-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-osd-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-namespace-client-key-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-namespace-client-key-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-storage-keys-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-storage-keys-generator
[storage] 2018/04/28 07:50:39 getting release history for "ceph"
[tiller] 2018/04/28 07:50:39 uninstall: Deleting ceph
[tiller] 2018/04/28 07:50:39 executing 2 pre-delete hooks for ceph
[kube] 2018/04/28 07:50:39 building resources from manifest
[kube] 2018/04/28 07:50:39 creating 1 resource(s)
[kube] 2018/04/28 07:50:39 Watching for changes to Job ceph-namespace-client-key-cleaner-qwkee with timeout of 5m0s
[kube] 2018/04/28 07:50:39 Add/Modify event for ceph-namespace-client-key-cleaner-qwkee: ADDED
[kube] 2018/04/28 07:50:39 ceph-namespace-client-key-cleaner-qwkee: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
[kube] 2018/04/28 07:50:40 Add/Modify event for ceph-namespace-client-key-cleaner-qwkee: MODIFIED
[kube] 2018/04/28 07:50:40 ceph-namespace-client-key-cleaner-qwkee: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
[kube] 2018/04/28 07:50:42 Add/Modify event for ceph-namespace-client-key-cleaner-qwkee: MODIFIED
[kube] 2018/04/28 07:50:42 building resources from manifest
[kube] 2018/04/28 07:50:42 creating 1 resource(s)
[kube] 2018/04/28 07:50:42 Watching for changes to Job ceph-storage-admin-key-cleaner-zemvx with timeout of 5m0s
[kube] 2018/04/28 07:50:42 Add/Modify event for ceph-storage-admin-key-cleaner-zemvx: ADDED
[kube] 2018/04/28 07:50:42 ceph-storage-admin-key-cleaner-zemvx: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
[kube] 2018/04/28 07:50:42 Add/Modify event for ceph-storage-admin-key-cleaner-zemvx: MODIFIED
[kube] 2018/04/28 07:50:42 ceph-storage-admin-key-cleaner-zemvx: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
[kube] 2018/04/28 07:50:45 Add/Modify event for ceph-storage-admin-key-cleaner-zemvx: MODIFIED
[tiller] 2018/04/28 07:50:45 hooks complete for pre-delete ceph
[storage] 2018/04/28 07:50:45 updating release "ceph.v1"
[kube] 2018/04/28 07:50:45 Starting delete for "ceph-mon" Service
[kube] 2018/04/28 07:50:45 Starting delete for "ceph-rgw" Service
[kube] 2018/04/28 07:50:45 Starting delete for "ceph-mds-keyring-generator" Job
[kube] 2018/04/28 07:50:45 Using reaper for deleting "ceph-mds-keyring-generator"
[kube] 2018/04/28 07:50:47 Starting delete for "ceph-mgr-keyring-generator" Job
[kube] 2018/04/28 07:50:47 Using reaper for deleting "ceph-mgr-keyring-generator"
[kube] 2018/04/28 07:50:49 Starting delete for "ceph-mon-keyring-generator" Job
[kube] 2018/04/28 07:50:49 Using reaper for deleting "ceph-mon-keyring-generator"
[kube] 2018/04/28 07:50:52 Starting delete for "ceph-rgw-keyring-generator" Job
[kube] 2018/04/28 07:50:52 Using reaper for deleting "ceph-rgw-keyring-generator"
[kube] 2018/04/28 07:50:54 Starting delete for "ceph-osd-keyring-generator" Job
[kube] 2018/04/28 07:50:54 Using reaper for deleting "ceph-osd-keyring-generator"
[kube] 2018/04/28 07:50:56 Starting delete for "ceph-namespace-client-key-generator" Job
[kube] 2018/04/28 07:50:56 Using reaper for deleting "ceph-namespace-client-key-generator"
[kube] 2018/04/28 07:50:58 Starting delete for "ceph-storage-keys-generator" Job
[kube] 2018/04/28 07:50:58 Using reaper for deleting "ceph-storage-keys-generator"
[kube] 2018/04/28 07:51:00 Starting delete for "ceph-mds" Deployment
[kube] 2018/04/28 07:51:00 Using reaper for deleting "ceph-mds"
[kube] 2018/04/28 07:51:03 Starting delete for "ceph-mgr" Deployment
[kube] 2018/04/28 07:51:03 Using reaper for deleting "ceph-mgr"
[kube] 2018/04/28 07:51:06 Starting delete for "ceph-mon-check" Deployment
[kube] 2018/04/28 07:51:06 Using reaper for deleting "ceph-mon-check"
[kube] 2018/04/28 07:51:09 Starting delete for "ceph-rbd-provisioner" Deployment
[kube] 2018/04/28 07:51:09 Using reaper for deleting "ceph-rbd-provisioner"
[kube] 2018/04/28 07:51:12 Starting delete for "ceph-rgw" Deployment
[kube] 2018/04/28 07:51:12 Using reaper for deleting "ceph-rgw"
[kube] 2018/04/28 07:51:15 Starting delete for "ceph-mon" DaemonSet
[kube] 2018/04/28 07:51:15 Using reaper for deleting "ceph-mon"
[kube] 2018/04/28 07:51:58 Starting delete for "ceph-osd-dev-sda" DaemonSet
[kube] 2018/04/28 07:51:58 Using reaper for deleting "ceph-osd-dev-sda"
[kube] 2018/04/28 07:52:58 Starting delete for "ceph-rbd" StorageClass
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-bin-clients" ConfigMap
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-bin" ConfigMap
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-etc" ConfigMap
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-templates" ConfigMap
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-keystone-user-rgw" Secret
2018/04/28 07:52:59 uninstall: Failed deletion of "ceph": no objects visited
[tiller] 2018/04/28 07:52:59 error: object not found, skipping delete
[tiller] 2018/04/28 07:52:59 executing 2 post-delete hooks for ceph
[tiller] 2018/04/28 07:52:59 hooks complete for post-delete ceph
[tiller] 2018/04/28 07:52:59 purge requested for ceph
[storage] 2018/04/28 07:52:59 deleting release "ceph.v1"
[tiller] 2018/04/28 07:54:07 preparing install for ceph
[storage] 2018/04/28 07:54:07 getting release history for "ceph"
[tiller] 2018/04/28 07:54:07 rendering ceph chart using values
2018/04/28 07:54:07 info: manifest "ceph/templates/job-ks-endpoints.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/configmap-bin-ks.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/job-bootstrap.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/secret-keystone.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/job-ceph-commands.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/daemonset-osd.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/job-ks-service.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/job-ks-user.yaml" is empty. Skipping.
[tiller] 2018/04/28 07:54:07 performing install for ceph
[tiller] 2018/04/28 07:54:07 executing 2 pre-install hooks for ceph
[tiller] 2018/04/28 07:54:07 hooks complete for pre-install ceph
[storage] 2018/04/28 07:54:07 getting release history for "ceph"
[storage] 2018/04/28 07:54:07 creating release "ceph.v1"
[kube] 2018/04/28 07:54:07 building resources from manifest
[kube] 2018/04/28 07:54:07 creating 22 resource(s)
[tiller] 2018/04/28 07:54:09 executing 2 post-install hooks for ceph
[tiller] 2018/04/28 07:54:09 hooks complete for post-install ceph
[storage] 2018/04/28 07:54:09 updating release "ceph.v1"
[storage] 2018/04/28 07:54:09 getting last revision of "ceph"
[storage] 2018/04/28 07:54:09 getting release history for "ceph"
[kube] 2018/04/28 07:54:09 Doing get for Secret: "ceph-keystone-user-rgw"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Secret/ceph-keystone-user-rgw
[kube] 2018/04/28 07:54:09 Doing get for ConfigMap: "ceph-bin-clients"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/ConfigMap/ceph-bin-clients
[kube] 2018/04/28 07:54:09 Doing get for ConfigMap: "ceph-bin"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/ConfigMap/ceph-bin
[kube] 2018/04/28 07:54:09 Doing get for ConfigMap: "ceph-etc"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/ConfigMap/ceph-etc
[kube] 2018/04/28 07:54:09 Doing get for ConfigMap: "ceph-templates"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/ConfigMap/ceph-templates
[kube] 2018/04/28 07:54:09 Doing get for StorageClass: "ceph-rbd"
[kube] 2018/04/28 07:54:09 get relation pod of object: /StorageClass/ceph-rbd
[kube] 2018/04/28 07:54:09 Doing get for Service: "ceph-mon"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Service/ceph-mon
[kube] 2018/04/28 07:54:09 Doing get for Service: "ceph-rgw"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Service/ceph-rgw
[kube] 2018/04/28 07:54:09 Doing get for DaemonSet: "ceph-mon"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/DaemonSet/ceph-mon
[kube] 2018/04/28 07:54:09 Doing get for DaemonSet: "ceph-osd-dev-sda"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/DaemonSet/ceph-osd-dev-sda
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-mds"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Deployment/ceph-mds
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-mgr"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Deployment/ceph-mgr
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-mon-check"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Deployment/ceph-mon-check
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-rbd-provisioner"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Deployment/ceph-rbd-provisioner
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-rgw"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Deployment/ceph-rgw
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-osd-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-osd-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-mds-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-mds-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-mon-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-mon-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-rgw-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-rgw-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-mgr-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-mgr-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-namespace-client-key-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-namespace-client-key-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-storage-keys-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-storage-keys-generator
foxundermoon commented
system centos CentOS Linux release 7.4.1708 (Core)
install by vmware
sudo fdisk -l
[sudo] password for k8s:
Disk /dev/sda: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000aae00
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 104857599 51379200 8e Linux LVM
Disk /dev/mapper/centos-root: 50.5 GB, 50457477120 bytes, 98549760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-swap: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Thu Apr 19 03:44:34 2018
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 0 0
UUID=fdf34de5-5810-4b0e-a7fe-4b92c03cf582 /boot xfs defaults 0 0
#/dev/mapper/centos-swap swap swap defaults 0 0
- values config
network:
public: 192.168.0.0/16
cluster: 192.168.0.0/16
port:
mon: 6789
rgw: 8088
enable_zap_and_potentially_lose_data: false
osd_devices:
- name: dev-sda
device: /dev/mapper/centos-root
# journal: /dev/sdf
zap: "1"
ceph_mgr_enabled_modules:
- restful
- status
- dashboard
# You can configure your mgr modules
# below. Each module has its own set
# of key/value. Refer to the doc
# above for more info. For example:
ceph_mgr_modules_config:
dashboard:
port: 7000
localpool:
failure_domain: host
subtree: rack
pg_num: "128"
num_rep: "3"
min_size: "2"
storageclass:
provision_storage_class: true
provisioner: ceph.com/rbd
name: ceph-rbd
monitors: null
pool: rbd
admin_id: admin
admin_secret_name: pvc-ceph-conf-combined-storageclass
admin_secret_namespace: ceph
user_id: admin
user_secret_name: pvc-ceph-client-key
image_format: "2"
image_features: layeringwhmzsu commented
I met that error before when I install it after uninstall a "failed helm install ceph before PR #54" . So I doubt that there maybe some files were not cleaned by the "helm delete --purge ". So I did a clean installation started from all new VMs and kubernetes cluster, finally got a successful installation of ceph.
whmzsu commented
BTW: It seems that you config the OSD device to use the OS device ??? I think you should add one or more bland disk for OSD device, NOT the OS root device
foxundermoon commented
@whmzsu thank you.
i deleted the /var/log/ceph /var/lib/ceph-helm
also clean the k8s all source under ceph namespace.
whmzsu commented
@foxundermoon I try to install the ceph again after uninstall the success installation, delete the ceph-helm dir , and got the same error like your. So I think there must be something not cleared by the helm while uninstalling.
praparn commented
We also found this issue on Kubernetes 1.9.2. As we try all is clear all helm environment/reinital k8s farm and try more than 3 times still issue this
foxundermoon commented
@whmzsu still issue. i am reinstall on other 3 servers.
still meet the error
Normal SuccessfulMountVolume 54m kubelet, bj-h03-11-cm.getui MountVolume.SetUp succeeded for volume "default-token-xp4fc"
Warning FailedMount 54m (x3 over 54m) kubelet, bj-h03-11-cm.getui MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found
Warning FailedMount 54m (x4 over 54m) kubelet, bj-h03-11-cm.getui MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found
Warning FailedMount 54m (x4 over 54m) kubelet, bj-h03-11-cm.getui MountVolume.SetUp failed for volume "ceph-bootstrap-osd-keyring" : secrets "ceph-bootstrap-osd-keyring" not found
Warning FailedMount 54m (x4 over 54m) kubelet, bj-h03-11-cm.getui MountVolume.SetUp failed for volume "ceph-bootstrap-mds-keyring" : secrets "ceph-bootstrap-mds-keyring" not found
Warning FailedMount 54m (x4 over 54m) kubelet, bj-h03-11-cm.getui MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secrets "ceph-client-admin-keyring" not found
Normal Pulled 48m kubelet, bj-h03-11-cm.getui Successfully pulled image "docker.io/kolla/ubuntu-source-kubernetes-entrypoint:4.0.0"
githubfoam commented
I am getting same problem.Problem is "/etc/ceph/ceph.mon.keyring" I do not have such a path "/etc/ceph" or "var/lib/ceph". How can I all do this with a VM ? I do not want to run a host with minikube.
Installation (Kubernetes + Helm)
minikube version: v0.27.0
helm version Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
stat: can't stat '/var/log/ceph/ceph.audit.log': No such file or directory
container "osd-activate-pod" in pod "ceph-osd-dev-sdd-pzqck" is waiting to start: PodInitializing
MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found
Back-off restarting failed container
{ "name": "ceph-mon-keyring", "readOnly": true, "mountPath": "/etc/ceph/ceph.mon.keyring", "subPath": "ceph.mon.keyring" },
pasikon commented
@githubfoam I have exactly same issue
githubfoam commented
pasikon commented
@githubfoam yes, 27.0
githubfoam commented
not merged yet I guess.
Fix keyring does not exist
ksingh7 commented
Encountered with the same issue in discussion
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 1m (x3 over 1m) kubelet, server-6 MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found
Warning FailedMount 1m (x4 over 1m) kubelet, server-6 MountVolume.SetUp failed for volume "ceph-bootstrap-osd-keyring" : secrets "ceph-bootstrap-osd-keyring" not found
Warning FailedMount 1m (x5 over 1m) kubelet, server-6 MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found
Warning FailedMount 1m (x6 over 1m) kubelet, server-6 MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secrets "ceph-client-admin-keyring" not found
Warning FailedMount 1m (x6 over 1m) kubelet, server-6 MountVolume.SetUp failed for volume "ceph-bootstrap-mds-keyring" : secrets "ceph-bootstrap-mds-keyring" not found
Normal Pulled 50s kubelet, server-6 Container image "docker.io/kolla/ubuntu-source-kubernetes-entrypoint:4.0.0" already present on machine
When i tried for the first time everything worked fine. As soon as i deleted the cluster using helm delete --purge ceph --timeout 1000 --debug and retried helm install --name=ceph local/ceph --namespace=ceph -f ~/ceph-overrides.yaml i started getting these errors.
Also i do not have directories like /etc/ceph /var/lib/ceph /var/log/ceph /var/lib/ceph-helm.
Need help to get find
A) Clean way to delete the cluster
B) Fix this issue.
@rootfs any suggestions ?
ksingh7 commented
@githubfoam i am running the fix you mentioned in your last comment.
ksingh7 commented
@foxundermoon @githubfoam did you ever managed to get this working. Whats the clean way to purge the cluster ?
sunx commented
You must remove the partitions on target devices before installing a new ceph cluster.
xforze commented
Hi All,
I have exactly the same issue with Kubernetes 1.11.2. Is there still no fix or workaround for this ?
@sunx The osd discs/partitions are untouched as the osd Pod are not coming up.
Cheers!
mamoit commented
Just use fdisk to create a new partition table.
xforze commented
The OSD Pods still not coming up, a "kubectl describe pod xxxxxx" shows:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 11m (x5 over 11m) kubelet, k8s-ceph3 MountVolume.SetUp failed for volume "ceph-bootstrap-mds-keyring" : secrets "ceph-bootstrap-mds-keyring" not found
Warning FailedMount 11m (x5 over 11m) kubelet, k8s-ceph3 MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secrets "ceph-client-admin-keyring" not found
Warning FailedMount 11m (x5 over 11m) kubelet, k8s-ceph3 MountVolume.SetUp failed for volume "ceph-bootstrap-osd-keyring" : secrets "ceph-bootstrap-osd-keyring" not found
Warning FailedMount 11m (x5 over 11m) kubelet, k8s-ceph3 MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found
Warning FailedMount 11m (x5 over 11m) kubelet, k8s-ceph3 MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found
Warning BackOff 1m (x41 over 10m) kubelet, k8s-ceph3 Back-off restarting failed container
But all of these Secrets are existing in the same namespace.
Amos-85 commented
same issue for me in K8S v1.10.5 / RHEL7.5 / KERNEL 3.10.0.
osd-prepare-pod with mounting event exceptions like @xforze .
any ideas?
elvinasp commented
same issue for me in K8S v1.10.5 / RHEL7.5 / KERNEL 3.10.0.
osd-prepare-pod with mounting event exceptions like @xforze .
Same issue occured on K8S v1.11.2 / Ubuntu 18.04. However after I deleted pods, they mounted secrets successfully.
After that I have stumbled upon other error related to reinstall. At fault was storage disks with old information from previous install. @mamoit incorrectly suggested to use fdisk, where you should use gdisk to wipe/recreate partitions! Cephs expects GPT partition table and fdisk does not properly handle that.
To verify, check logs from init container:
kubectl -n ceph logs ceph-osd-dev-sdc-XXXX -c osd-prepare-pod
I was getting the following error:
create_partition: Creating journal partition num 2 size 5120 on /dev/sdc
command_check_call: Running command: /sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:5341d5ed-b6eb-4bcd-8e35-1620d85ef776 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdc
Caution: invalid main GPT header, but valid backup; regenerating main header
from backup!
Invalid partition data!
/sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:5341d5ed-b6eb-4bcd-8e35-1620d85ef776 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdc' failed with status code 2
After I have recreated GPT partition table (option 2) with gdisk /dev/sdc, pod successfully initialized.
Amos-85 commented
cool @elvinasp !
I should give a try for that.
I moved to rook.io project to deploy ceph.
danielxolsson commented
I had the exact same issue. The last piece in the cleanup puzzle was to delete dangling docker volumes on all nodes:
docker volume rm $(docker volume ls -qf dangling=true)
neuhalje commented
Purge Ceph
The following will remove all remains of Ceph (after that I could install). This assumes that the nodename is reachable via ssh.
helm del --purge ceph
kubectl delete namespaces ceph
while read host; do
echo $host; ssh -n $host -l root -- rm -rf /var/lib/ceph-helm /etc/ceph
done < <(kubectl get nodes -o name|sed 's!^node/!!')I also rebooted all nodes (not sure if this is a must).
Verify that kubernetes is gone:
kubectl get all --all-namespaces -o name|grep cephEDIT: Still fails :-(
Deleting the OSD pods (works!)
After deleting all OSD pods the cluster came up:
kubectl delete pod -n ceph ceph-osd-dev-vdb-m28wr
...robin-2016 commented
the same problem.
when I run make:
find: ‘secrets’: No such file or directory
xinghui-hou commented
I have same problem.
ceph-rgw-keyring-generator job create pod ，alway run failed 。
This causes other created pods to fail all the time
I've tried purge twice, but it's the same problem
tx19980520 commented
I have same problem.
ceph-rgw-keyring-generatorjob create pod ，alway run failed 。
This causes other created pods to fail all the time
I've tried purge twice, but it's the same problem
the same, @xinghui-hou, did you find the solution?
alter commented
Hello, kubernetes 1.15.5 and the same things here...
Does this project(I mean k8s chart) is still alive?
sudo-undefined commented
Last commit was on 26:th June 2019.
ceph-helm is not deployable on 1.16+ and as we can see have problems in earlier versions as well
I'm going for rook-ceph instead
githubfoam commented
@githubfoam i am running the fix you mentioned in your last comment.
I did not try but only way possible is to git clone/code/compile the solution and run it locally I guess.The solution seems merged.You can see the details of root cause analysis in the link I refer to. Manually typing "auth" in the "ceph.conf" could be last resort before pulling the source.