UDP Proxy Protocol not functioning
Opened this issue · 19 comments
Hi All,
It appears that there is some issue with the proxy protocol implemented for UDP as my server recieves this response when trying to proxy.
[2024-11-18 00:00:04 Local] [192.168.200.1:37825] [UDPPROXY] System.ArgumentNullException: Value cannot be null. (Parameter 'address')
at System.ArgumentNullException.Throw(String paramName)
at DnsServerCore.Dns.DnsServer.ReadUdpRequestAsync(Socket udpListener, DnsTransportProtocol protocol)
[2024-11-18 00:00:04 Local] [192.168.200.1:37825] [UDPPROXY] System.IO.InvalidDataException: The stream does not contain PROXY protocol header.
at TechnitiumLibrary.Net.ProxyProtocol.ProxyProtocolStream.CreateAsServerAsync(Stream baseStream, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\ProxyProtocol\ProxyProtocolStream.cs:line 138
at DnsServerCore.Dns.DnsServer.ReadUdpRequestAsync(Socket udpListener, DnsTransportProtocol protocol) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\DnsServer.cs:line 409
But when I use nginx, it works fine. I did raise this with the provider and they also confirmed my findings
TechnitiumSoftware/DnsServer#1107
TCP works fine for proxying as I'm also utilizing that, but I would like to cover both sides of it.
The thing I see on the caddy side when this is run is the below errors.
{"level":"debug","ts":1731887445.6253898,"logger":"layer4","msg":"matching","remote":"172.253.247.61:39349","error":"consumed all prefetched bytes","matcher":"layer4.matchers.dns","matched":false}
{"level":"debug","ts":1731887445.6254132,"logger":"layer4","msg":"prefetched","remote":"172.253.247.61:39349","bytes":48}
{"level":"debug","ts":1731887445.6254213,"logger":"layer4","msg":"matching","remote":"172.253.247.61:39349","matcher":"layer4.matchers.dns","matched":true}
{"level":"debug","ts":1731887446.644327,"logger":"layer4","msg":"matching","remote":"172.68.209.70:19458","error":"consumed all prefetched bytes","matcher":"layer4.matchers.dns","matched":false}
{"level":"debug","ts":1731887446.64437,"logger":"layer4","msg":"prefetched","remote":"172.68.209.70:19458","bytes":48}
{"level":"debug","ts":1731887446.6443822,"logger":"layer4","msg":"matching","remote":"172.68.209.70:19458","matcher":"layer4.matchers.dns","matched":true}
{"level":"debug","ts":1731887446.6450343,"logger":"layer4","msg":"matching","remote":"172.68.209.70:19483","error":"consumed all prefetched bytes","matcher":"layer4.matchers.dns","matched":false}
{"level":"debug","ts":1731887446.6450565,"logger":"layer4","msg":"prefetched","remote":"172.68.209.70:19483","bytes":52}
{"level":"debug","ts":1731887446.6450644,"logger":"layer4","msg":"matching","remote":"172.68.209.70:19483","matcher":"layer4.matchers.dns","matched":true}
This is what I have set up for the block
layer4 {
tcp/:53 {
@tdns dns
route @tdns {
proxy {
upstream tcp/ext-dns:538
proxy_protocol v2
}
}
}
udp/:53 {
@udns dns
route @udns {
proxy {
proxy_protocol v2
upstream {
dial udp/ext-dns:538
}
}
}
}
}
And this is what I set up with nginx to confirm that it was its definitely an issue with caddy-l4
stream {
log_format proxy '$remote_addr [$time_local] '
'$protocol $status $bytes_sent $bytes_received '
'$session_time "$upstream_addr" '
'"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';
upstream dns {
server ext-dns:538;
}
server {
listen 53 udp;
proxy_pass dns;
proxy_protocol on;
}
access_log /config/log/nginx/access-dns.log proxy buffer=32k;
}
cc/ @WeidiDeng
I think the layer4 proxy handler just does not intend to support proxy protocol with udp.
I am not sure how proxy protocol over udp is supposed to work, but it probably has to send the header in front of each packet instead of only once.
Is there a plan to fix this bug?
@erroltuparker can you try xcaddy build --with github.com/mholt/caddy-l4=github.com/WeidiDeng/caddy-l4@udp-pp to see if it's fixed?
hi @WeidiDeng, i just tried it and still the same errors
Application endpoint
[2025-01-12 22:12:42 UTC] [192.168.86.5:41931] [UDPPROXY] System.IO.InvalidDataException: The stream does not contain PROXY protocol header.
at TechnitiumLibrary.Net.ProxyProtocol.ProxyProtocolStream.CreateAsServerAsync(Stream baseStream, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\ProxyProtocol\ProxyProtocolStream.cs:line 138
at DnsServerCore.Dns.DnsServer.ReadUdpRequestAsync(Socket udpListener, DnsTransportProtocol protocol) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\DnsServer.cs:line 415
[2025-01-12 22:12:42 UTC] [192.168.86.5:41931] [UDPPROXY] System.FormatException: An invalid IP address was specified.
---> System.Net.Sockets.SocketException (22): Invalid argument
--- End of inner exception stack trace ---
at TechnitiumLibrary.Net.ProxyProtocol.ProxyProtocolStream.ParseVersion1(String value) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\ProxyProtocol\ProxyProtocolStream.cs:line 245
at TechnitiumLibrary.Net.ProxyProtocol.ProxyProtocolStream.CreateAsServerAsync(Stream baseStream, CancellationToken cancellationToken)
at DnsServerCore.Dns.DnsServer.ReadUdpRequestAsync(Socket udpListener, DnsTransportProtocol protocol) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\DnsServer.cs:line 415
Caddy Layer4
{"level":"debug","ts":1736719962.905754,"logger":"layer4","msg":"matching","remote":"206.189.140.177:48228","error":"consumed all prefetched bytes","matcher":"layer4.matchers.dns","matched":false}
{"level":"debug","ts":1736719962.905816,"logger":"layer4","msg":"prefetched","remote":"206.189.140.177:48228","bytes":39}
{"level":"debug","ts":1736719962.9058948,"logger":"layer4","msg":"matching","remote":"206.189.140.177:48228","matcher":"layer4.matchers.dns","matched":true}
{"level":"debug","ts":1736719982.9061859,"logger":"layer4","msg":"matching","remote":"206.189.140.177:34585","error":"consumed all prefetched bytes","matcher":"layer4.matchers.dns","matched":false}
{"level":"debug","ts":1736719982.9062274,"logger":"layer4","msg":"prefetched","remote":"206.189.140.177:34585","bytes":28}
{"level":"debug","ts":1736719982.9062378,"logger":"layer4","msg":"matching","remote":"206.189.140.177:34585","matcher":"layer4.matchers.dns","matched":true}
{"level":"debug","ts":1736720002.9057178,"logger":"layer4","msg":"connection stats","remote":"206.189.140.177:48228","read":78,"written":0,"duration":39.999970735}
{"level":"debug","ts":1736720022.9075844,"logger":"layer4","msg":"connection stats","remote":"206.189.140.177:34585","read":56,"written":0,"duration":40.001406982}
@erroltuparker How do you deploy DnsServer using docker? I'll need to test it locally to figure out what's wrong.
@WeidiDeng Can the following example be used for testing?
{
"apps": {
"layer4": {
"servers": {
"tcpsni": {
"listen": [":443"],
"routes": [{
"match": [{
"tls": {}
}],
"handle": [{
"handler": "proxy",
"proxy_protocol": "v2",
"upstreams": [{
"dial": ["127.0.0.1:8443"]
}]
}]
}]
},
"udpsni": {
"listen": ["udp/:443"],
"routes": [{
"match": [{
"quic": {}
}],
"handle": [{
"handler": "proxy",
"proxy_protocol": "v2",
"upstreams": [{
"dial": ["udp/127.0.0.1:8443"]
}]
}]
}]
}
}
},
"http": {
"servers": {
"srvh3": {
"listen": ["127.0.0.1:8443"],
"listener_wrappers": [{
"wrapper": "proxy_protocol",
"allow": ["127.0.0.1/32"]
},
{
"wrapper": "tls"
}],
"routes": [{
"handle": [{
"handler": "headers",
"response": {
"set": {
"Alt-Svc": ["h3=\":443\"; ma=2592000"],
"Strict-Transport-Security": ["max-age=31536000; includeSubDomains; preload"]
}
}
},
{
"handler": "file_server",
"root": "/var/www/html"
}]
}],
"tls_connection_policies": [{
"match": {
"sni": ["xx.yy"]
}
}]
}
}
},
"tls": {
"certificates": {
"automate": ["xx.yy"]
},
"automation": {
"policies": [{
"issuers": [{
"module": "acme",
"email": "your@mail.com"
},
{
"module": "acme",
"ca": "https://acme.zerossl.com/v2/DV90",
"email": "your@mail.com"
}]
}]
}
}
}
}
@WeidiDeng as im using unraid its basically this run command (cleaned up some of the extra bits)
docker run
-d
--name='dns'
--net='bond0'
--ip='192.168.86.1'
--cpuset-cpus='16,17,18,19,20,21,22,23'
--pids-limit 2048
-v '/mnt/user/appdata/technitium-dnsserver/config':'/etc/dns/config':'rw'
-v '/mnt/user/appdata/technitium-dnsserver/':'/etc/dns/':'rw'
--hostname=dns
--sysctl net.ipv6.conf.all.disable_ipv6=0
--sysctl net.ipv6.conf.eth0.use_tempaddr=2
--restart=unless-stopped 'technitium/dns-server'
As its on an actual network, all the ports are available to caddy.
Then I have Technitium configured as below
Then the caddyfile as configured below
layer4 {
tcp/:53 {
@tdns {
dns
}
route @tdns {
proxy {
upstream tcp/dns:538
proxy_protocol v2
}
}
}
udp/:53 {
@udns {
dns
}
route @udns {
proxy {
proxy_protocol v2
upstream udp/dns:538
}
}
}
udp/:853 {
@quic {
quic sni dns.example.com
}
route @quic {
proxy {
proxy_protocol v2
upstream udp/dns:538
}
}
}
tcp/:853 {
@dot {
tls sni dns.example.com
}
route @dot {
tls
proxy {
upstream tcp/dns:538
proxy_protocol v2
}
}
}
}
It's actually upstream's bug, i.e. https://github.com/mastercactapus/proxyprotocol.
When writing a proxyprotocol header, for udp, the local address is &net.UDPAddr{IP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, Port:53, Zone:""} if using your configuration. This ip address is net.IPv6zero which can't be shorten to ipv4 version. The remote address, however, contains an IP address that can be shortened to ipv4.
The upstream library can't handle this case, and it will write an invalid proxy protocol header. And, this header will parse successfully by this library, even though the type is wrong.
This library can only handle tcp address for v1 proxy protocol
caddy core already uses another proxyprotocol library (better maintained and more commonly used), I'll check if that one is working successfully and use that instead the current one. But that will be a big refactor.
update: the new library doesn't support proxyprotocol for network other than tcp either.
@erroltuparker can you try xcaddy build --with github.com/mholt/caddy-l4=github.com/WeidiDeng/caddy-l4@udp-pp again to see if it's fixed?
@WeidiDeng still the same issue but please see the results between proxy_protocol v1 and proxy_protocol v2
note: 192.168.86.5 is caddy
proxy_protocol v2
[2025-01-14 08:23:37 UTC] [192.168.86.5:36809] [UDPPROXY] System.IO.InvalidDataException: The stream does not contain PROXY protocol header.
at TechnitiumLibrary.Net.ProxyProtocol.ProxyProtocolStream.CreateAsServerAsync(Stream baseStream, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\ProxyProtocol\ProxyProtocolStream.cs:line 138
at DnsServerCore.Dns.DnsServer.ReadUdpRequestAsync(Socket udpListener, DnsTransportProtocol protocol) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\DnsServer.cs:line 415
[2025-01-14 08:23:37 UTC] [192.168.86.5:36809] [UDPPROXY] System.ArgumentNullException: Value cannot be null. (Parameter 'address')
at System.ArgumentNullException.Throw(String paramName)
at System.Net.IPEndPoint..ctor(IPAddress address, Int32 port)
at DnsServerCore.Dns.DnsServer.ReadUdpRequestAsync(Socket udpListener, DnsTransportProtocol protocol) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\DnsServer.cs:line 416
proxy_protocol v1
[2025-01-14 08:29:59 UTC] [192.168.86.5:59897] [UDPPROXY] System.IO.InvalidDataException: The stream does not contain PROXY protocol header.
at TechnitiumLibrary.Net.ProxyProtocol.ProxyProtocolStream.CreateAsServerAsync(Stream baseStream, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\ProxyProtocol\ProxyProtocolStream.cs:line 138
at DnsServerCore.Dns.DnsServer.ReadUdpRequestAsync(Socket udpListener, DnsTransportProtocol protocol) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\DnsServer.cs:line 415
[2025-01-14 08:29:59 UTC] [192.168.86.5:59897] [UDPPROXY] System.FormatException: An invalid IP address was specified.
---> System.Net.Sockets.SocketException (22): Invalid argument
--- End of inner exception stack trace ---
at TechnitiumLibrary.Net.ProxyProtocol.ProxyProtocolStream.ParseVersion1(String value) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\ProxyProtocol\ProxyProtocolStream.cs:line 245
at TechnitiumLibrary.Net.ProxyProtocol.ProxyProtocolStream.CreateAsServerAsync(Stream baseStream, CancellationToken cancellationToken)
at DnsServerCore.Dns.DnsServer.ReadUdpRequestAsync(Socket udpListener, DnsTransportProtocol protocol) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\DnsServer.cs:line 415
proxy_protocol v1 can not be fixed, both upstream library doesn't support this use case. Only tcp is supported.
I think nginx can extract the exact address using this type of syscall. This requires rewriting packet connection handling to get the exact local address. For golang this local address is 0.0.0.0, which may be rejected by DnsServer.
https://pkg.go.dev/golang.org/x/net@v0.34.0/ipv4#PacketConn.ReadFrom
https://pkg.go.dev/golang.org/x/net@v0.34.0/ipv6#PacketConn.ReadFrom
I won't have time to work on this for now.
@erroltuparker can you try xcaddy build --with github.com/mholt/caddy-l4=github.com/WeidiDeng/caddy-l4@udp-pp again to see if it's fixed?
No need to test v1 since it isn't supported by upstream at all.
Hi @WeidiDeng, still same errors.
(btw, thanks for looking into this)
Hello @WeidiDeng,
I tried your build and it seems to work nicely with my Technitium deployment. Testing on port 54 and Podman:
# allow access to host's 538 (UDP-PROXY port exposed by Technitium)
podman run --rm --net pasta:-U,538 -v ./Caddyfile:/etc/caddy/Caddyfile -p 54:54/udp caddy-l4:udp-ppCaddyfile
{
log {
level debug
}
layer4 {
udp/:54 {
@udns dns
route @udns {
proxy {
proxy_protocol v2
upstream udp/127.0.0.1:538
}
}
}
}
}
Tested command:
dog -U example.com @<dns-server-ip>:54
@erroltuparker it seems that Technitium only logs error. The original requesting IPs shows up as expected in Query Logs so please check that too.
This is amazing development! It seems like we are on track towards HTTP3/QUIC with accurate IPs.
Hello @skedastically @WeidiDeng,
But the program crashes when using the following example.
{
log {
level DEBUG
}
layer4 {
udp/:443 {
@qsni quic sni xx.yy
route @qsni {
proxy {
upstream udp/127.0.0.1:2443
}
}
route {
proxy {
#proxy_protocol v2
upstream udp/127.0.0.1:8443
}
}
}
}
}
root@C20240207094726:~# journalctl -u caddy --no-pager
........
Jan 20 01:55:34 C20240207094726 caddy[524]: panic: runtime error: invalid memory address or nil pointer dereference
Jan 20 01:55:34 C20240207094726 caddy[524]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x157b284]
Jan 20 01:55:34 C20240207094726 caddy[524]: goroutine 92 [running]:
Jan 20 01:55:34 C20240207094726 caddy[524]: github.com/mholt/caddy-l4/modules/l4proxy.(*packetProxyProtocolConn).Write(0xc000158080, {0xc0006a8000, 0x546, 0x2000})
Jan 20 01:55:34 C20240207094726 caddy[524]: github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/modules/l4proxy/proxy.go:210 +0x44
Jan 20 01:55:34 C20240207094726 caddy[524]: io.(*teeReader).Read(0xc000158120, {0xc0006a8000, 0xc0d3e8?, 0x2000})
Jan 20 01:55:34 C20240207094726 caddy[524]: io/io.go:630 +0x76
Jan 20 01:55:34 C20240207094726 caddy[524]: io.discard.ReadFrom({}, {0x1eed700,
0xc000158120})
Jan 20 01:55:34 C20240207094726 caddy[524]: io/io.go:666 +0x6d
Jan 20 01:55:34 C20240207094726 caddy[524]: io.copyBuffer({0x1eed740, 0x2c0e800}, {0x1eed700, 0xc000158120}, {0x0, 0x0, 0x0})
Jan 20 01:55:34 C20240207094726 caddy[524]: io/io.go:415 +0x151
Jan 20 01:55:34 C20240207094726 caddy[524]: io.Copy(...)
Jan 20 01:55:34 C20240207094726 caddy[524]: io/io.go:388
Jan 20 01:55:34 C20240207094726 caddy[524]: github.com/mholt/caddy-l4/modules/l4proxy.(*Handler).proxy.func2()
Jan 20 01:55:34 C20240207094726 caddy[524]: github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/modules/l4proxy/proxy.go:340 +0x59
Jan 20 01:55:34 C20240207094726 caddy[524]: created by github.com/mholt/caddy-l4/modules/l4proxy.(*Handler).proxy in goroutine 77
Jan 20 01:55:34 C20240207094726 caddy[524]: github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead/modules/l4proxy/proxy.go:337 +0x369
Jan 20 01:55:34 C20240207094726 systemd[1]: caddy.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 20 01:55:34 C20240207094726 systemd[1]: caddy.service: Failed with result 'exit-code'.
Hi @WeidiDeng , The mentioned BUG has been fixed, but a new BUG was found in the example tests.
{
log {
level DEBUG
}
layer4 {
tcp/:443 {
@tsni tls sni zz.yy
route @tsni {
proxy {
proxy_protocol v2
upstream unix/@uds7443.sock
}
}
route {
proxy {
proxy_protocol v2
upstream tcp/127.0.0.1:8443
}
}
}
udp/:443 {
@qsni quic sni xx.yy
route @qsni {
proxy {
upstream udp/127.0.0.1:2443
}
}
route {
proxy {
#proxy_protocol v2
upstream udp/127.0.0.1:8443
}
}
}
}
servers 127.0.0.1:8443 {
listener_wrappers {
proxy_protocol {
allow 127.0.0.1/32
}
tls
}
}
}
:8443, xx.zz:8443 {
bind 127.0.0.1
header {
Alt-Svc "h3=\":443\"; ma=2592000"
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
}
file_server {
root /var/www/html
}
}
BUG：
For unix/@uds7443.sock, proxy protocol transmission cannot be enabled. Everything works normally when both the sender and receiver have proxy protocol disabled.
unix/@uds7443.sock Error Log:
........
2025/01/23 04:14:51.315 �[35mDEBUG�[0m layer4.matchers.tls matched {"remote": "171.21.70.185:45450", "server_name": "hy.xx.yy"}
2025/01/23 04:14:51.315 �[35mDEBUG�[0m layer4 matching {"remote": "171.21.70.185:45450", "matcher": "layer4.matchers.tls", "matched": true}
2025/01/23 04:14:51.315 �[35mDEBUG�[0m layer4.handlers.proxy dial upstream {"remote": "171.21.70.185:45450", "upstream": "@uds7443.sock"}
2025/01/23 04:14:51.315 �[35mDEBUG�[0m layer4 matching {"remote": "171.21.70.185:45464", "error": "consumed all prefetched bytes", "matcher": "layer4.matchers.tls", "matched": false}
2025/01/23 04:14:51.330 �[35mDEBUG�[0m layer4 prefetched {"remote": "171.21.70.185:45468", "bytes": 563}
2025/01/23 04:14:51.330 �[35mDEBUG�[0m layer4 matching {"remote": "171.21.70.185:45468", "matcher": "layer4.matchers.tls", "matched": false}
........
@WeidiDeng Hi, there’s a bug when forwarding with UDS. Master Xi, please fix it when you have time.