Use the nfs service DNS name instead of IP to specify the PV

Hello,

We've had a disaster recovery on our cluster and we had to re-provision the nfs server provisioner.

When we reprovisioned the server, it created a new service which got a new IP address, which made all the PVs invalid.
We restored the functionality by adding a new ad-hoc service in which we specified the service IP to correspond to the PVs ones. We now need this wart in the cluster.

I was wondering if we could use the nfs service name instead of the IP when provisioning the PVs ?

Right now, when the PV is created, the IP is placed:

Source:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    10.222.45.230
    Path:      /export/pvc-fd579ce4-6203-4eb8-a5eb-30436d8cf52f
    ReadOnly:  false

See here:

nfs-ganesha-server-and-external-provisioner/pkg/volume/provision.go

Line 451 in 09caa21

return service.Spec.ClusterIP, nil

where we return the service IP instead of the name.

I'm quite sure there's a good reason for doing it via IP, but this is limiting when there's a disaster recoveery. Any idea/comment ?

I think your suggestion makes sense. I think that using an IP address is indeed a bit unreliable in case someone recreates the service. But after trying to fix this for a while, I found out the reason why IP addresses are being used.

I discovered this: the actual NFS mount is not created by the pod or container, but takes place on the host operating system. Under the hood, the "mount -t nfs" command is thus given on the node, and this system is (normally) unaware of the DNS names used inside the Kubernetes cluster. The host OS uses its own DNS resolver, and not the one running in Kubernetes, and therefore names like "servicename.namespace" or "servicename.namespace.svc.cluster.local" cannot be resolved, which causes the mount command to fail. In my opinion this is a Kubernetes design flaw, because it would logically make so much more sense when such names could be used as NFS server name.

Anyway, there are a number of solutions, and none of them is very elegant:

Accept it (haha) and just make sure (or hope) that you never have to recreate the service, so that the IP always stay the same.
Use a hardcoded IP address in the service definition. The "Service" resource allows this. Recreating the service will then always give the same IP address. See: https://stackoverflow.com/questions/62262656/how-to-define-a-static-clusterip-in-kubernetes. This is a bit problematic if your services are created automatically, but if you do it by hand possible.
If you do want to use DNS names, you need some additional work. Maybe you're lucky and your cluster already is configured like this (some clusters are), but otherwise you have to update the /etc/resolv.conf on each node and make it point to 10.96.0.10 (which is normally the coredns IP). I don't know if this is clever though as it creates a circular dependency (the nodes rely on kubernetes and kubernetes relies on the nodes). And even if you do this, it still requires some (small!) changes to the sourcecode of the provisioner to make it work, see pull request #92, and the nfs-provisioner process must be started with the "-server-hostname=service.namespace.svn.cluster.local" command line option.

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen

Mark this issue or PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.