Mounting CephFS is giving an Error.

[saitejar]

Request for Help.
I am using ceph-helm, also running mds with it.
ceph -s shows that everything is up and running.

I am unable to mount ceph fs to a pod.

I am using this ceph-test-fs.yaml

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: cephfs
  name: ceph-cephfs-test
  namespace: ceph
spec:
  nodeSelector:
    node-type: storage
  containers:
  - name: cephfs-rw
    image: busybox
    command:
    - sh
    - -c
    - while true; do sleep 1; done
    volumeMounts:
    - mountPath: "/mnt/cephfs"
      name: cephfs
  volumes:
  - name: cephfs
    cephfs:
      monitors:
#This only works if you have skyDNS resolveable from the kubernetes node. Otherwise you must manually put in one or more mon pod ips.
      - ceph-mon.ceph:6789
      user: admin
      secretRef:
        name: ??

I am not sure what keyring to use to be able to mount.
and also I did try all the secrets generated by ceph-helm deployment.

Can anyone help me with mounting ceph fs?

Here is the error log:

MountVolume.SetUp failed for volume "cephfs" : CephFS: mount failed: mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/7d82c944-e01f-11e7-bf03-001c42d61047/volumes/kubernetes.io~cephfs/cephfs --scope -- mount -t ceph -o name=admin,secretfile=/etc/ceph/admin.secret ceph-mon.ceph:6789:/ /var/lib/kubelet/pods/7d82c944-e01f-11e7-bf03-001c42d61047/volumes/kubernetes.io~cephfs/cephfs Output: Running scope as unit run-10734.scope. mount: wrong fs type, bad option, bad superblock on ceph-mon.ceph:6789:/, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so.

[rootfs]

• which kubernetes release you are using?
• on that node, are you able to ping ceph-mon.ceph (and check if your host's DNS is set to kube-dns service ip)?
• what do you see in syslog and dmesg?

[saitejar]

@rootfs
Yes, I can ping ceph-mon.ceph
In dmesg, I see.

[29119.038942] libceph: resolve 'ceph-mon.ceph' (ret=0): 10.211.55.195:0
[29119.038947] libceph: bad option at 'secret'

Looks like I am using a wrong secret. But I am unsure which secret to use with admin user.
These are the secrets that ceph-helm deployment has created.

pvc-ceph-conf-combined-storageclass
ceph-bootstrap-mgr-keyring
ceph-client-admin-keyring
ceph-bootstrap-mds-keyring
ceph-bootstrap-osd-keyring
ceph-bootstrap-rgw-keyring
ceph-mon-keyring
pvc-ceph-client-key
ceph-keystone-user-rgw

[rootfs]

how did you provision cephfs PV? it looks mount picked up secret file over secret

[saitejar]

@rootfs, you are right. I was giving the key file name instead of keyfile. I am able to mount now. Thanks a lot for your help.