rook/nfs

mount.nfs connection refused

meirhazonAnyVision opened this issue · 6 comments

meirhazonAnyVision commented

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

Expected behavior:

How to reproduce it (minimal and precise):

Environment:

  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Cloud provider or hardware configuration:
  • Rook version (use rook version inside of a Rook Pod):
  • Kubernetes version (use kubectl version):
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):
meirhazonAnyVision commented

Hello,
I am using rook nfs and I am getting the following, what is the cause?

Thanks

ok-nfs-pv
Feb 6 22:19:49 ceph-2 kubelet: Output: Running scope as unit run-17427.scope.
Feb 6 22:19:49 ceph-2 kubelet: mount.nfs: Connection refused
Feb 6 22:19:49 ceph-2 kubelet: E0206 22:19:49.177344 28654 nestedpendingoperations.go:267] Operation for ""kubernetes.io/nfs/b5860e94-2a5c-11e9-85d8-42010a840fd5-rook-nfs-pv" ("b5860e94-2a5c-11e9-85d8-42010a840fd5")" failed. No retries permitted until 2019-02-06 22:19:50.177295237 +0000 UTC m=+45650.582595199 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume "rook-nfs-pv" (UniqueName: "kubernetes.io/nfs/b5860e94-2a5c-11e9-85d8-42010a840fd5-rook-nfs-pv") pod "wordpress-mysql-5d499f5dcd-5wbjc" (UID: "b5860e94-2a5c-11e9-85d8-42010a840fd5") : mount failed: exit status 32\nMounting command: systemd-run\nMounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/b5860e94-2a5c-11e9-85d8-42010a840fd5/volumes/kubernetes.ionfs/rook-nfs-pv --scope -- mount -t nfs -o vers=4.1 10.98.14.12:/nfs-ceph-claim /var/lib/kubelet/pods/b5860e94-2a5c-11e9-85d8-42010a840fd5/volumes/kubernetes.ionfs/rook-nfs-pv\nOutput: Running scope as unit run-17427.scope.\nmount.nfs: Connection refused\n\n"
Feb 6 22:19:50 ceph-2 systemd: Started Kubernetes transient mount for /var/lib/kubelet/pods/b5860e94-2a5c-11e9-85d8-42010a840fd5/volumes/kubernetes.io~nfs/rook-nfs-pv.

Warning FailedMount 2m20s kubelet, ceph-3 MountVolume.SetUp failed for volume "rook-nfs-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/3dfb8666-2a5c-11e9-85d8-42010a840fd5/volumes/kubernetes.ionfs/rook-nfs-pv --scope -- mount -t nfs -o vers=4.1 10.98.14.12:/nfs-ceph-claim /var/lib/kubelet/pods/3dfb8666-2a5c-11e9-85d8-42010a840fd5/volumes/kubernetes.ionfs/rook-nfs-pv
Output: Running scope as unit run-26565.scope.
mount.nfs: Connection refused
Warning FailedMount 2m12s (x5 over 11m) kubelet, ceph-3 Unable to mount volumes for pod "rook-nfs-0_default(3dfb8666-2a5c-11e9-85d8-42010a840fd5)": timeout expired waiting for volumes to attach or mount for pod "default"/"rook-nfs-0". list of unmounted volumes=[nfs-ceph-claim]. list of unattached volumes=[nfs-ceph-claim nfs-ganesha-config default-token-9ns5x]
Warning FailedMount 2s kubelet, ceph-3 MountVolume.SetUp failed for volume "rook-nfs-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/3dfb8666-2a5c-11e9-85d8-42010a840fd5/volumes/kubernetes.ionfs/rook-nfs-pv --scope -- mount -t nfs -o vers=4.1 10.98.14.12:/nfs-ceph-claim /var/lib/kubelet/pods/3dfb8666-2a5c-11e9-85d8-42010a840fd5/volumes/kubernetes.ionfs/rook-nfs-pv
Output: Running scope as unit run-26987.scope.
mount.nfs: Connection refused

meirhazonAnyVision commented

Hi, some Env info,

Running in Google Cloud

[root@ceph-1 ceph]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
[root@ceph-1 ceph]# uname -a
Linux ceph-1 3.10.0-957.5.1.el7.x86_64 rook/rook#1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@ceph-1 ceph]# rpm -qa|grep kube
kubeadm-1.13.3-0.x86_64
kubernetes-cni-0.6.0-0.x86_64
kubectl-1.13.3-0.x86_64
kubelet-1.13.3-0.x86_64
[root@ceph-1 ceph]# rpm -qa|grep nfs
libnfsidmap-0.25-19.el7.x86_64
nfs-utils-1.3.0-0.61.el7.x86_64
[root@ceph-1 ceph]# systemctl status nfs
● nfs-server.service - NFS server and services
Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)
Active: active (exited) since Wed 2019-02-06 20:16:10 UTC; 3h 35min ago
Process: 13444 ExecStartPost=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl restart gssproxy ; fi (code=exited, status=0/SUCCESS)
Process: 13428 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
Process: 13425 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
Main PID: 13428 (code=exited, status=0/SUCCESS)
Tasks: 0
Memory: 0B
CGroup: /system.slice/nfs-server.service

Feb 06 20:16:10 ceph-1 systemd[1]: Starting NFS server and services...
Feb 06 20:16:10 ceph-1 systemd[1]: Started NFS server and services.

***ceph health` in the Rook Ceph toolbox HANGS...

[root@ceph-1 ceph]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
ceph-1 Ready master 14h v1.13.3
ceph-2 Ready 14h v1.13.3
ceph-3 Ready 14h v1.13.3
ceph-4 Ready 14h v1.13.3

[root@ceph-1 ceph]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default rook-nfs-0 0/1 ContainerCreating 0 102m
default wordpress-mysql-5d499f5dcd-5wbjc 0/1 ContainerCreating 0 99m
kube-system coredns-86c58d9df4-ctg59 1/1 Running 3 14h
kube-system coredns-86c58d9df4-t5tlf 1/1 Running 2 14h
kube-system etcd-ceph-1 1/1 Running 2 9h
kube-system kube-apiserver-ceph-1 1/1 Running 27 9h
kube-system kube-controller-manager-ceph-1 1/1 Running 2 14h
kube-system kube-flannel-ds-amd64-b299p 1/1 Running 1 14h
kube-system kube-flannel-ds-amd64-s7l4t 1/1 Running 2 14h
kube-system kube-flannel-ds-amd64-tjljw 1/1 Running 0 14h
kube-system kube-flannel-ds-amd64-x5fgf 1/1 Running 1 14h
kube-system kube-proxy-24g2b 1/1 Running 0 14h
kube-system kube-proxy-h8slx 1/1 Running 1 14h
kube-system kube-proxy-sgc28 1/1 Running 1 14h
kube-system kube-proxy-wfvsb 1/1 Running 3 14h
kube-system kube-scheduler-ceph-1 1/1 Running 27 9h
rook-ceph-system rook-ceph-agent-2g265 1/1 Running 0 6h36m
rook-ceph-system rook-ceph-agent-gztsj 1/1 Running 0 6h36m
rook-ceph-system rook-ceph-agent-t55v2 1/1 Running 0 6h36m
rook-ceph-system rook-ceph-operator-76cf7f88f-xmgjb 1/1 Running 0 6h36m
rook-ceph-system rook-discover-7cxvr 1/1 Running 0 6h36m
rook-ceph-system rook-discover-fm2h4 1/1 Running 0 6h36m
rook-ceph-system rook-discover-svzkc 1/1 Running 0 6h36m
rook-ceph rook-ceph-mon-a-656959c8b7-zvf59 1/1 Running 0 6h36m
rook-ceph rook-ceph-mon-d-75dd67c4b-vqkkw 1/1 Running 0 6h29m
rook-ceph rook-ceph-mon-f-86d98fbf7d-vd5ps 1/1 Running 0 6h22m
rook-ceph rook-ceph-tools-76c7d559b6-h5sqv 1/1 Running 0 6h18m
rook-nfs-system rook-nfs-operator-8b6d9d9ff-47ncx 1/1 Running 0 6h45m

root@ceph-1 ceph]# kubectl get pv --all-namespaces
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
rook-nfs-pv 2Gi RWX Retain Bound default/nfs-ceph-claim rook-ceph-block 102m
[root@ceph-1 ceph]# kubectl get pvc --all-namespaces
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
default nfs-ceph-claim Bound rook-nfs-pv 2Gi RWX rook-ceph-block 103m
[root@ceph-1 ceph]#

Thanks so much

travisn commented

@meirhazonAnyVision It appears the ceph cluster was not initialized correctly, so you will not be able to mount the storage until it is healthy. Your main symptom is that the only pods running in the rook-ceph namespace are the mons. This help topic may help explain why the mons are not forming quorum. After they form quorum, you'll see osd pods start, and you should see ceph status complete successfully in the toolbox.

teramucho commented

Hello,
I am using rook nfs and I am getting the following, what is the cause?

Thanks

ok-nfs-pv
Feb 6 22:19:49 ceph-2 kubelet: Output: Running scope as unit run-17427.scope.
Feb 6 22:19:49 ceph-2 kubelet: mount.nfs: Connection refused
Feb 6 22:19:49 ceph-2 kubelet: E0206 22:19:49.177344 28654 nestedpendingoperations.go:267] Operation for ""kubernetes.io/nfs/b5860e94-2a5c-11e9-85d8-42010a840fd5-rook-nfs-pv" ("b5860e94-2a5c-11e9-85d8-42010a840fd5")" failed. No retries permitted until 2019-02-06 22:19:50.177295237 +0000 UTC m=+45650.582595199 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume "rook-nfs-pv" (UniqueName: "kubernetes.io/nfs/b5860e94-2a5c-11e9-85d8-42010a840fd5-rook-nfs-pv") pod "wordpress-mysql-5d499f5dcd-5wbjc" (UID: "b5860e94-2a5c-11e9-85d8-42010a840fd5") : mount failed: exit status 32\nMounting command: systemd-run\nMounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/b5860e94-2a5c-11e9-85d8-42010a840fd5/volumes/kubernetes.ionfs/rook-nfs-pv --scope -- mount -t nfs -o vers=4.1 10.98.14.12:/nfs-ceph-claim /var/lib/kubelet/pods/b5860e94-2a5c-11e9-85d8-42010a840fd5/volumes/kubernetes.ionfs/rook-nfs-pv\nOutput: Running scope as unit run-17427.scope.\nmount.nfs: Connection refused\n\n"
Feb 6 22:19:50 ceph-2 systemd: Started Kubernetes transient mount for /var/lib/kubelet/pods/b5860e94-2a5c-11e9-85d8-42010a840fd5/volumes/kubernetes.io~nfs/rook-nfs-pv.

Warning FailedMount 2m20s kubelet, ceph-3 MountVolume.SetUp failed for volume "rook-nfs-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/3dfb8666-2a5c-11e9-85d8-42010a840fd5/volumes/kubernetes.ionfs/rook-nfs-pv --scope -- mount -t nfs -o vers=4.1 10.98.14.12:/nfs-ceph-claim /var/lib/kubelet/pods/3dfb8666-2a5c-11e9-85d8-42010a840fd5/volumes/kubernetes.ionfs/rook-nfs-pv
Output: Running scope as unit run-26565.scope.
mount.nfs: Connection refused
Warning FailedMount 2m12s (x5 over 11m) kubelet, ceph-3 Unable to mount volumes for pod "rook-nfs-0_default(3dfb8666-2a5c-11e9-85d8-42010a840fd5)": timeout expired waiting for volumes to attach or mount for pod "default"/"rook-nfs-0". list of unmounted volumes=[nfs-ceph-claim]. list of unattached volumes=[nfs-ceph-claim nfs-ganesha-config default-token-9ns5x]
Warning FailedMount 2s kubelet, ceph-3 MountVolume.SetUp failed for volume "rook-nfs-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/3dfb8666-2a5c-11e9-85d8-42010a840fd5/volumes/kubernetes.ionfs/rook-nfs-pv --scope -- mount -t nfs -o vers=4.1 10.98.14.12:/nfs-ceph-claim /var/lib/kubelet/pods/3dfb8666-2a5c-11e9-85d8-42010a840fd5/volumes/kubernetes.ionfs/rook-nfs-pv
Output: Running scope as unit run-26987.scope.
mount.nfs: Connection refused

i have same issue...

it looks like onwership error, but i don't know exactly...

stale commented

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale commented

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.