pivotal-cf/terraforming-azure

PKS cannot be used if vnet has custom DNS

Closed this issue · 1 comments

aegershman commented

I've come across this attempting split-horizon DNS & associating the DNS servers at the vnet layer. Want to share in case it helps others.

  • By default, an Azure vnet uses Azure-provided DNS. This gives the vnet built-in hostname resolution for all the VMs created. E.g., a VM with a hostname of 527a4d0c-bc61-4913-9e80-cb0cec6f4b63 can be resolved by other VMs within the vnet simply by querying for 527a4d0c-bc61-4913-9e80-cb0cec6f4b63, and Azure DNS will resolve it's IP.
  • When you query for a direct interaction with a pod such as kubectl logs or kubectl exec, or helm list, etc., the kubeapi server has to resolve the host of the node the pod is living on. E.g., if the masters know pod-0 is on host 527a4d0c-bc61-4913-9e80-cb0cec6f4b63, and you query for kubectl logs pod-0, the api server will attempt to resolve the host of 527a4d0c-bc61-4913-9e80-cb0cec6f4b63 to an IP...
  • But if Azure-provided DNS isn't being used, and thus the VMs have no dynamic DNS which allows them to resolve one another (in this case, allows the master VMs to resolve the worker VM which has the pod running on it), then you get "host not found" errors.
  • IF you associate a DNS server to the vnet, then Azure-provided DNS backs off entirely, and you lose this built-in hostname resolution. Even if your custom DNS servers are VMs within the vnet; even if you have your DNS servers recurse to the Azure-provided DNS, etc.-- if you associate a custom DNS entry at the vnet layer, Azure-provided DNS backs off, and PKS will basically break.

I'm still working through the best topology for split-horizon DNS, but the current plan is to have the custom BIND9 DNS servers run within the vnet as normal VMs, and update BOSH's cloud-config to have all VMs within a network use the BIND9 servers for it's DNS. Then the BIND9 servers will forward certain domains to our onprem DNS, and all other requests onto Azure-provided DNS at 168.63.129.16, which will still perform hostname resolution.

That's the working theory, at least. Wanted to put it out there for others to see; if anyone looks at this & sees something off or that I'm misunderstanding/misconfiguring, please let me know 👍

What's fun about this is that you won't this particular error during cluster provisioning or smoke tests, you only see them when you try to interact with the pods directly on their hosts 🎉

See also:

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-name-resolution-for-vms-and-role-instances

When you use your own name resolution solution, this suffix is not supplied to virtual machines because the suffix interferes with other DNS architectures... When you are using your own name resolution solution, this suffix is not supplied to VMs because it interferes with other DNS architectures (like domain-joined scenarios). Instead, Azure provides a non-functioning placeholder (reddog.microsoft.com).

https://docs.microsoft.com/en-us/azure/virtual-network/manage-virtual-network#change-dns-servers

All VMs that are connected to the virtual network register with the DNS servers that you specify for the virtual network. They also use the specified DNS server for name resolution. Each network interface (NIC) in a VM can have its own DNS server settings. If a NIC has its own DNS server settings, they override the DNS server settings for the virtual network.

svrc commented

I built the original Azure PKS support a few years ago with Microsoft's help. There's a long history here.
cloudfoundry/bosh-agent#174
cloudfoundry/bosh-dns-release#31

I'm going to add this to PKS docs soon but there are a couple of workarounds here

  • For most cases of BYO DNS on Azure PKS, I got the BOSH DNS team to make BOSH agent IDs to be DNS resolvable as of v1.15. However, they do it under a private top level domain. This requires a search entry in the worker nodes resolv.conf. This isn't in the upstream product yet. Workaround add-on instructions here for any PKS foundation: https://github.com/svrc-pivotal/pks-azure-dns-nodes

  • Another vetted approach we've used is to use BOSH DNS handlers for private corporate zones. https://bosh.io/docs/dns/#additional-handlers , which are supported in Ops Man. This way, Azure DNS is the only DNS server and the DHCP DDNS hostname registration works fine against *.internal.cloudapp.net. But since all the workers go through BOSH DNS, we can insert handlers to direct certain zones to corporate DNS servers.