DNS is no longer working

Question

DNS is no longer working

Closed this issue 8 months ago · 9 comments

Once I ran following command:

docker run -v ovpn-data:/etc/openvpn -d --network host  --cap-add=NET_ADMIN my-openvpn-dev-ovpn-server

DNS is no longer working.
Or more precisely my ubuntu server is no longer able to resolve domain names.

root@nuc1:~# ping google.be
ping: google.be: Temporary failure in name resolution
root@nuc1:~# ping google.be
ping: google.be: Temporary failure in name resolution
root@nuc1:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=116 time=16.3 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=116 time=16.2 ms
``
`

Answer 1 · 2024-01-29T13:47:22.000Z

I see that it has created a bridge br0 that is bridging the 2 interfaces enp3s0 and tap0.

root@nuc1:~# brctl show
bridge name     bridge id               STP enabled     interfaces
br-4fd482291b4b         8000.0242f9bf320a       no              veth0807452
                                                        veth0ee19e3
                                                        ...
                                                        vethfb72f3a
br-91ce6ebfb546         8000.0242fe8324c5       no
br0             8000.94c691a2ad53       no              enp3s0
                                                        tap0
docker0         8000.0242e4877d39       no
root@nuc1:~#

The problem is that no DNS seems to be setup for the bridge interface:

root@nuc1:~# resolvectl status
Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (enp3s0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.2.1
       DNS Servers: 192.168.2.1
        DNS Domain: lan

Link 3 (wlp2s0) 
...
Link 50 (vethdd9167b)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 51 (tap0)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 52 (br0)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Answer 2 · 2024-01-29T14:46:22.000Z

FYI this is what is written in my container log file:

2024-01-29 12:57:34 OpenVPN 2.6.8 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
2024-01-29 12:57:34 library versions: OpenSSL 3.1.4 24 Oct 2023, LZO 2.10
2024-01-29 12:57:34 NOTE: when bridging your LAN adapter with the TAP adapter, note that the new bridge adapter will often take on its own IP address that is different from what the LAN adapter was previously set to
2024-01-29 12:57:34 Diffie-Hellman initialized with 2048 bit key
2024-01-29 12:57:34 CRL: loaded 1 CRLs from file /etc/openvpn/crl.pem
2024-01-29 12:57:34 ROUTE_GATEWAY 192.168.2.1/255.255.255.0 IFACE=br0 HWADDR=94:c6:91:a2:ad:53
2024-01-29 12:57:34 OpenVPN ROUTE: OpenVPN needs a gateway parameter for a --route option and no default was specified by either --route-gateway or --ifconfig options
2024-01-29 12:57:34 OpenVPN ROUTE: failed to parse/resolve route for host/network: 192.168.254.0
2024-01-29 12:57:34 TUN/TAP device tap0 opened
2024-01-29 12:57:34 Could not determine IPv4/IPv6 protocol. Using AF_INET
2024-01-29 12:57:34 Socket Buffers: R=[212992->212992] S=[212992->212992]
2024-01-29 12:57:34 UDPv4 link local (bound): [AF_INET][undef]:1196
2024-01-29 12:57:34 UDPv4 link remote: [AF_UNSPEC]
2024-01-29 12:57:34 GID set to nogroup
2024-01-29 12:57:34 UID set to nobody
2024-01-29 12:57:34 MULTI: multi_init called, r=256 v=256
2024-01-29 12:57:34 IFCONFIG POOL IPv4: base=192.168.2.200 size=21
2024-01-29 12:57:34 Initialization Sequence Completed

and this is my routing table when the openvpn server is running.

root@nuc1:~# ip route
default via 192.168.2.1 dev br0 
default via 192.168.2.1 dev enp3s0 proto dhcp src 192.168.2.11 metric 20100 
169.254.0.0/16 dev br0 scope link metric 1000 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.18.0.0/16 dev br-4fd482291b4b proto kernel scope link src 172.18.0.1 
172.19.0.0/16 dev br-91ce6ebfb546 proto kernel scope link src 172.19.0.1 linkdown 
192.168.2.0/24 dev br0 proto kernel scope link src 192.168.2.11 
192.168.2.0/24 dev enp3s0 proto kernel scope link src 192.168.2.11 metric 100 
root@nuc1:~#

and this is the routing table before I have started my openvpn server:

root@nuc1:~# ip route
default via 192.168.2.1 dev enp3s0 proto dhcp src 192.168.2.11 metric 100 
169.254.0.0/16 dev enp3s0 scope link metric 1000 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.18.0.0/16 dev br-4fd482291b4b proto kernel scope link src 172.18.0.1 
172.19.0.0/16 dev br-91ce6ebfb546 proto kernel scope link src 172.19.0.1 linkdown 
192.168.2.0/24 dev enp3s0 proto kernel scope link src 192.168.2.11 metric 100 
root@nuc1:~#

Answer 3 · 2024-01-29T18:57:01.000Z

Hm, I just tested the setup from blank in a new VM and the server came up, DNS still working.
Could you please post the contents of your /etc/openvpn/ovpn_env.sh file? I'm also missing the log output for setting up the bridge, that's the part before OpenVPN server is started. Could you please add that as well?

Setting up bridge
setting tap device to promiscous mode
setting eth device to promiscous mode
setting IP, subnet and broadcast address for bridge
setting mac address for bridge
checking if default gateway needs to be added for bridge
setting up IPTABLES
default via fe80::209:4fff:fe72:6d1a dev enp0s3 proto ra metric 100 pref medium
default via fe80::209:4fff:fe72:6d1a dev br0 proto ra metric 1024 expires 1772sec hoplimit 64 pref medium
Checking IPv6 Forwarding
Sysctl error for default forwarding, please run docker with '--sysctl net.ipv6.conf.default.forwarding=1'
Sysctl error for all forwarding, please run docker with '--sysctl net.ipv6.conf.all.forwarding=1'
Running 'openvpn --config /etc/openvpn/openvpn.conf --client-config-dir /etc/openvpn/ccd --crl-verify /etc/openvpn/crl.pem '
2024-01-29 18:30:31 DEPRECATED OPTION: --cipher set to 'AES-256-CBC' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305). OpenVPN ignores --cipher for cipher negotiations. 
2024-01-29 18:30:31 WARNING: Using --management on a TCP port WITHOUT passwords is STRONGLY discouraged and considered insecure
2024-01-29 18:30:31 OpenVPN 2.6.8 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
2024-01-29 18:30:31 library versions: OpenSSL 3.1.4 24 Oct 2023, LZO 2.10
2024-01-29 18:30:31 MANAGEMENT: TCP Socket listening on [AF_INET][undef]:5560
2024-01-29 18:30:31 NOTE: when bridging your LAN adapter with the TAP adapter, note that the new bridge adapter will often take on its own IP address that is different from what the LAN adapter was previously set to
2024-01-29 18:30:31 NOTE: your local LAN uses the extremely common subnet address 192.168.0.x or 192.168.1.x.  Be aware that this might create routing conflicts if you connect to the VPN server from public locations such as internet cafes that use the same subnet.
2024-01-29 18:30:31 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2024-01-29 18:30:31 Diffie-Hellman initialized with 2048 bit key
2024-01-29 18:30:31 CRL: loaded 1 CRLs from file /etc/openvpn/crl.pem
2024-01-29 18:30:31 TUN/TAP device tap0 opened
2024-01-29 18:30:31 Could not determine IPv4/IPv6 protocol. Using AF_INET
2024-01-29 18:30:31 Socket Buffers: R=[212992->212992] S=[212992->212992]
2024-01-29 18:30:31 UDPv4 link local (bound): [AF_INET][undef]:443
2024-01-29 18:30:31 UDPv4 link remote: [AF_UNSPEC]
2024-01-29 18:30:31 GID set to nogroup
2024-01-29 18:30:31 UID set to nobody
2024-01-29 18:30:31 MULTI: multi_init called, r=256 v=256
2024-01-29 18:30:31 IFCONFIG POOL IPv4: base=192.168.1.220 size=6
2024-01-29 18:30:31 Initialization Sequence Completed

root@DebianTest:/etc/openvpn# ping google.be
PING google.be(fra24s08-in-x03.1e100.net (2a00:1450:4001:82b::2003)) 56 data bytes
64 bytes from fra24s08-in-x03.1e100.net (2a00:1450:4001:82b::2003): icmp_seq=1 ttl=118 time=16.9 ms
64 bytes from fra24s08-in-x03.1e100.net (2a00:1450:4001:82b::2003): icmp_seq=2 ttl=118 time=17.2 ms
64 bytes from fra24s08-in-x03.1e100.net (2a00:1450:4001:82b::2003): icmp_seq=3 ttl=118 time=16.9 ms
64 bytes from fra24s08-in-x03.1e100.net (2a00:1450:4001:82b::2003): icmp_seq=4 ttl=118 time=17.5 ms

Edit: For the record: I'm using this on a barebone Debian install without NetworkManager, systemd-resolved and such. Not sure if/what difference that makes, never used Ubuntu Server before.

Answer 4 · 2024-01-29T22:36:55.000Z

my ovpn_env.sh:

declare -x OVPN_AUTH=
declare -x OVPN_BR_BR=br0
declare -x OVPN_BR_DHCP_END=192.168.2.220
declare -x OVPN_BR_DHCP_START=192.168.2.200
declare -x OVPN_BR_ETH_BROADCAST=192.168.2.255
declare -x OVPN_BR_ETH_GATEWAY=192.168.2.1
declare -x OVPN_BR_ETH_IF=enp3s0
declare -x OVPN_BR_ETH_IP=192.168.2.11
declare -x OVPN_BR_ETH_MAC=94:c6:91:a2:ad:53
declare -x OVPN_BR_ETH_SUBNET=255.255.255.0
declare -x OVPN_CIPHER=
declare -x OVPN_CLIENT_TO_CLIENT=
declare -x OVPN_CN=niamondo.be
declare -x OVPN_COMP_LZO=0
declare -x OVPN_DEFROUTE=1
declare -x OVPN_DEVICE=tap
declare -x OVPN_DEVICEN=0
declare -x OVPN_DISABLE_PUSH_BLOCK_DNS=0
declare -x OVPN_DNS=1
declare -x OVPN_DNS_SERVERS=([0]="8.8.8.8" [1]="8.8.4.4")
declare -x OVPN_ENV=/etc/openvpn/ovpn_env.sh
declare -x OVPN_EXTRA_CLIENT_CONFIG=()
declare -x OVPN_EXTRA_SERVER_CONFIG=()
declare -x OVPN_FRAGMENT=
declare -x OVPN_KEEPALIVE='10 60'
declare -x OVPN_MTU=
declare -x OVPN_NAT=0
declare -x OVPN_PORT=1196
declare -x OVPN_PROTO=udp
declare -x OVPN_PUSH=()
declare -x OVPN_ROUTES=([0]="192.168.254.0/24")
declare -x OVPN_SERVER=192.168.255.0/24
declare -x OVPN_SERVER_URL=udp://niaXXXXXX.be:1196
declare -x OVPN_TLS_CIPHER=
declare -x OVPN_USE_BRIDGE=1

The complete log:

 *  Executing task: docker logs --tail 1000 -f 41ab5044c49ecc61c4562329c13556598e1163ee68a3b80e800d995a69ad7e1a 

Identity added: /Users/jan/.ssh/id_rsa (jan@Jans-MBP.lan)
2024-01-29 22:35:54 TUN/TAP device tap0 opened
2024-01-29 22:35:54 Persist state set to: ON
Setting up bridge
setting tap device to promiscous mode
setting eth device to promiscous mode
setting IP, subnet and broadcast address for bridge
setting mac address for bridge
checking if default gateway needs to be added for bridge
setting up IPTABLES
Checking IPv6 Forwarding
Sysctl error for default forwarding, please run docker with '--sysctl net.ipv6.conf.default.forwarding=1'
Sysctl error for all forwarding, please run docker with '--sysctl net.ipv6.conf.all.forwarding=1'
Running 'openvpn --config /etc/openvpn/openvpn.conf --client-config-dir /etc/openvpn/ccd --crl-verify /etc/openvpn/crl.pem '
2024-01-29 22:35:54 Note: --cipher is not set. OpenVPN versions before 2.5 defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
2024-01-29 22:35:54 OpenVPN 2.6.8 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
2024-01-29 22:35:54 library versions: OpenSSL 3.1.4 24 Oct 2023, LZO 2.10
2024-01-29 22:35:54 NOTE: when bridging your LAN adapter with the TAP adapter, note that the new bridge adapter will often take on its own IP address that is different from what the LAN adapter was previously set to
2024-01-29 22:35:54 Diffie-Hellman initialized with 2048 bit key
2024-01-29 22:35:54 CRL: loaded 1 CRLs from file /etc/openvpn/crl.pem
2024-01-29 22:35:54 ROUTE_GATEWAY 192.168.2.1/255.255.255.0 IFACE=br0 HWADDR=94:c6:91:a2:ad:53
2024-01-29 22:35:54 OpenVPN ROUTE: OpenVPN needs a gateway parameter for a --route option and no default was specified by either --route-gateway or --ifconfig options
2024-01-29 22:35:54 OpenVPN ROUTE: failed to parse/resolve route for host/network: 192.168.254.0
2024-01-29 22:35:54 TUN/TAP device tap0 opened
2024-01-29 22:35:54 Could not determine IPv4/IPv6 protocol. Using AF_INET
2024-01-29 22:35:54 Socket Buffers: R=[212992->212992] S=[212992->212992]
2024-01-29 22:35:54 UDPv4 link local (bound): [AF_INET][undef]:1196
2024-01-29 22:35:54 UDPv4 link remote: [AF_UNSPEC]
2024-01-29 22:35:54 GID set to nogroup
2024-01-29 22:35:54 UID set to nobody
2024-01-29 22:35:54 MULTI: multi_init called, r=256 v=256
2024-01-29 22:35:54 IFCONFIG POOL IPv4: base=192.168.2.200 size=21
2024-01-29 22:35:54 Initialization Sequence Completed

Answer 5 · 2024-01-29T22:51:44.000Z

Note that restarting systemd-resolved (command systemctl restart systemd-resolved) fixed the issue:

root@nuc1:~# systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-01-29 15:50:44 CET; 7h ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 456 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 6844)
     Memory: 7.6M
        CPU: 8.585s
     CGroup: /system.slice/systemd-resolved.service
             └─456 /lib/systemd/systemd-resolved

Jan 29 15:50:44 nuc1 systemd[1]: Starting systemd-resolved.service - Network Name Resolution...
Jan 29 15:50:44 nuc1 systemd-resolved[456]: Positive Trust Anchors:
Jan 29 15:50:44 nuc1 systemd-resolved[456]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jan 29 15:50:44 nuc1 systemd-resolved[456]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-add>
Jan 29 15:50:44 nuc1 systemd-resolved[456]: Using system hostname 'nuc1'.
Jan 29 15:50:44 nuc1 systemd[1]: Started systemd-resolved.service - Network Name Resolution.
Jan 29 15:50:48 nuc1 systemd-resolved[456]: enp3s0: Bus client set search domain list to: lan
Jan 29 15:50:48 nuc1 systemd-resolved[456]: enp3s0: Bus client set default route setting: yes
Jan 29 15:50:48 nuc1 systemd-resolved[456]: enp3s0: Bus client set DNS server list to: 192.168.2.1
root@nuc1:~# systemctl restart systemd-resolved
root@nuc1:~# systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-01-29 23:48:47 CET; 1s ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 246471 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 6844)
     Memory: 2.7M
        CPU: 366ms
     CGroup: /system.slice/systemd-resolved.service
             └─246471 /lib/systemd/systemd-resolved

Jan 29 23:48:46 nuc1 systemd[1]: Starting systemd-resolved.service - Network Name Resolution...
Jan 29 23:48:47 nuc1 systemd-resolved[246471]: Positive Trust Anchors:
Jan 29 23:48:47 nuc1 systemd-resolved[246471]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jan 29 23:48:47 nuc1 systemd-resolved[246471]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in->
Jan 29 23:48:47 nuc1 systemd-resolved[246471]: Using system hostname 'nuc1'.
Jan 29 23:48:47 nuc1 systemd[1]: Started systemd-resolved.service - Network Name Resolution.
root@nuc1:~# ping www.google.be
PING www.google.be (142.251.36.3) 56(84) bytes of data.
64 bytes from ams15s44-in-f3.1e100.net (142.251.36.3): icmp_seq=1 ttl=116 time=15.9 ms
64 bytes from ams15s44-in-f3.1e100.net (142.251.36.3): icmp_seq=2 ttl=116 time=15.8 ms
64 bytes from ams15s44-in-f3.1e100.net (142.251.36.3): icmp_seq=3 ttl=116 time=16.4 ms

Answer 6 · 2024-01-30T06:29:15.000Z

Thanks, a couple more questions:

Are you able to connect to your VPN server even though host DNS is not working?
Did restarting systemd-resolved fix DNS on the host while the VPN server were still running?

I'll set up an Ubuntu Server machine and try to reproduce this issue.

Answer 7 · 2024-01-30T07:12:33.000Z

I can directly connect to VPN server.
I am not connecting to VPN server via VPN client.

Restarting systemd-resolved fixed the DNS on the host while the VPN server is running

Answer 8 · 2024-01-30T08:59:29.000Z

Note that it reports a temporary failure in name resolution

root@nuc1:~# ping google.be
ping: google.be: Temporary failure in name resolution

I noticed that it seems to fix itself after a couple of minutes.

Answer 9 · 2024-01-30T11:14:33.000Z

I have written:

I noticed that it seems to fix itself after a couple of minutes.

This is not true.
It is only true when I have enabled ipv6 forwarding in /etc/sysctl.conf file (net.ipv6.conf.all.forwarding=1)