Connection immediately closed
gili-gili opened this issue · 13 comments
gili-gili commented
Symptoms
Unable to connect with naiveproxy, connections immediately closed after establishing.
Config
Caddy
{
"apps": {
"http": {
"http_port": 80,
"https_port": 443,
"servers": {
"server": {
"listen": [":80", ":443"],
"routes": [
{
"match": [
{
"host": ["example.com"]
}
],
"handle": [
{
"auth_credentials": [
"base64encoded"
],
"handler": "forward_proxy",
"hide_ip": true,
"hide_via": true,
"probe_resistance": {}
},
{
"handler": "file_server",
"root": "/path/to/index",
"index_names": ["index.html"]
}
]
}, // Other routes
],
"tls_connection_policies": [
{
"alpn": ["h2", "h3"], // "http/1.1" can cause weird ssl problems idk why, i should keep it default
"protocol_min": "tls1.3", // Very secure
"protocol_max": "tls1.3"
}
],
"automatic_https": {
"disable_redirects": true
},
"trusted_proxies": {
"source": "static",
"ranges": [
// IP list of cloudflare endpoints
]
},
"protocols": ["h1", "h2", "h3"] // Useless definition
}
}
},
"tls": {
"automation": {
"policies": [
{
"issuers": [
{
"email": "admin@example.com",
"module": "acme",
"ca": "https://acme-v02.api.letsencrypt.org/directory"
},
{
"email": "admin@example.com",
"module": "zerossl",
"ca": "https://acme.zerossl.com/v2/DV90"
}
],
"key_type": "p256"
}
]
}
}
}
}
Naiveproxy
{
"listen": "socks://127.0.0.1:8080",
"proxy": "https://example:example@example.com",
"log": ""
}
Version
naiveproxy: v123.0.6312.40-1
caddy: build in 4/17/2024
Logs
Client logs
[0418/015433.796:INFO:naive_proxy_bin.cc(484)] Proxying via https://example.com
[0418/015433.797:INFO:naive_proxy_bin.cc(686)] Listening on socks://127.0.0.1:8080
[0418/015444.741:INFO:naive_connection.cc(273)] Connection 1 to 1.1.1.1:443
[0418/015445.268:INFO:naive_proxy_delegate.cc(137)] [https://example.com:443] negotiated padding type: None
[0418/015445.269:INFO:naive_proxy.cc(184)] Connection 1 closed: OK
[0418/015445.269:INFO:naive_connection.cc(273)] Connection 2 to 1.1.1.1:443
[0418/015445.432:INFO:naive_proxy.cc(184)] Connection 2 closed: OK
[0418/015453.771:INFO:naive_connection.cc(273)] Connection 3 to www.google.com:443
[0418/015453.922:INFO:naive_proxy.cc(184)] Connection 3 closed: OK
[0418/015453.922:INFO:naive_connection.cc(273)] Connection 4 to www.google.com:443
[0418/015454.069:INFO:naive_proxy.cc(184)] Connection 4 closed: OK
[0418/015455.635:INFO:naive_connection.cc(273)] Connection 5 to www.google.com:443
[0418/015455.789:INFO:naive_proxy.cc(184)] Connection 5 closed: OK
[0418/015455.790:INFO:naive_connection.cc(273)] Connection 7 to www.google.com:443
[0418/015455.801:INFO:naive_connection.cc(273)] Connection 6 to www.google.com:443
[0418/015455.884:INFO:naive_connection.cc(273)] Connection 8 to www.google.com:443
[0418/015455.937:INFO:naive_proxy.cc(184)] Connection 7 closed: OK
[0418/015455.953:INFO:naive_proxy.cc(184)] Connection 6 closed: OK
[0418/015455.953:INFO:naive_connection.cc(273)] Connection 9 to www.google.com:443
[0418/015456.009:INFO:naive_connection.cc(273)] Connection 10 to www.google.com:443
[0418/015456.032:INFO:naive_proxy.cc(184)] Connection 8 closed: OK
[0418/015456.033:INFO:naive_connection.cc(273)] Connection 11 to www.google.com:443
[0418/015456.100:INFO:naive_proxy.cc(184)] Connection 9 closed: OK
[0418/015456.120:INFO:naive_connection.cc(273)] Connection 12 to www.google.com:443
[0418/015456.156:INFO:naive_proxy.cc(184)] Connection 10 closed: OK
[0418/015456.156:INFO:naive_connection.cc(273)] Connection 13 to www.google.com:443
[0418/015456.180:INFO:naive_proxy.cc(184)] Connection 11 closed: OK
[0418/015456.220:INFO:naive_connection.cc(273)] Connection 14 to www.google.com:443
[0418/015456.270:INFO:naive_proxy.cc(184)] Connection 12 closed: OK
[0418/015456.270:INFO:naive_connection.cc(273)] Connection 15 to www.google.com:443
[0418/015456.307:INFO:naive_proxy.cc(184)] Connection 13 closed: OK
[0418/015456.346:INFO:naive_connection.cc(273)] Connection 16 to www.google.com:443
[0418/015456.369:INFO:naive_proxy.cc(184)] Connection 14 closed: OK
[0418/015456.369:INFO:naive_connection.cc(273)] Connection 17 to www.google.com:443
[0418/015456.417:INFO:naive_proxy.cc(184)] Connection 15 closed: OK
[0418/015456.494:INFO:naive_proxy.cc(184)] Connection 16 closed: OK
[0418/015456.494:INFO:naive_connection.cc(273)] Connection 18 to www.google.com:443
[0418/015456.516:INFO:naive_proxy.cc(184)] Connection 17 closed: OK
[0418/015456.641:INFO:naive_proxy.cc(184)] Connection 18 closed: OK
[0418/015456.816:INFO:naive_connection.cc(273)] Connection 19 to 8.8.8.8:443
[0418/015456.963:INFO:naive_proxy.cc(184)] Connection 19 closed: OK
[0418/015456.964:INFO:naive_connection.cc(273)] Connection 20 to 8.8.8.8:443
[0418/015457.113:INFO:naive_proxy.cc(184)] Connection 20 closed: OK
cURL logs
$ curl -v -L -x socks5://127.0.0.1:8080 1.1.1.1
* Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
* SOCKS5 connect to IPv4 1.1.1.1:80 (locally resolved)
* SOCKS5 request granted.
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: 1.1.1.1
> User-Agent: curl/8.0.1
> Accept: */*
>
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server
Caddy server logs
Jan 01 01:01:04 Server caddy[1234]: {"level":"info","ts":1000000000.123,"logger":"tls","msg":"finished cleaning storage units"}
Jan 01 01:01:20 Server caddy[1234]: {"level":"debug","ts":1000000000.123,"logger":"events","msg":"event","name":"tls_get_certificate","id":"123456789-12312-12312","origin":"tls","data":{"client_hello":{"CipherSuites":[60138,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"example.com","SupportedCurves":[43690,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[14906,772,771],"RemoteAddr":{"IP":"1234:5678:90ab::cdef","Port":12345,"Zone":""},"LocalAddr":{"IP":"1234:5678:90ab::cdef","Port":443,"Zone":""}}}}
Jan 01 01:01:20 Server caddy[1234]: {"level":"debug","ts":1000000000.123,"logger":"tls.handshake","msg":"choosing certificate","identifier":"example.com","num_choices":1}
Jan 01 01:01:20 Server caddy[1234]: {"level":"debug","ts":1000000000.123,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"example.com","subjects":["example.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"1234567890"}
Jan 01 01:01:20 Server caddy[1234]: {"level":"debug","ts":1000000000.123,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"1234:5678:90ab::cdef","remote_port":"12345","subjects":["example.com"],"managed":true,"expiration":1718545254,"hash":"1234567890"}
tcpdump on server (inbound)
// I'm poor in tcpdump dont laugh at me
$ sudo tcpdump -i eth0 host CLIENT and dst port 443
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:34:01.332590 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [S], seq 842308868, win 64800, options [mss 1432,nop,wscale 8,nop,nop,sackOK], length 0
12:34:01.475130 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [.], ack 2283624645, win 1029, length 0
12:34:01.476124 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 0:536, ack 1, win 1029, length 536
12:34:01.620536 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [.], ack 2865, win 1029, length 0
12:34:01.624589 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 536:600, ack 3392, win 1027, length 64
12:34:01.625586 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 600:692, ack 3392, win 1027, length 92
12:34:01.625715 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 692:857, ack 3392, win 1027, length 165
12:34:01.767123 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 857:888, ack 3453, win 1026, length 31
12:34:01.767916 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [.], ack 3519, win 1026, length 0
12:34:01.771748 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 888:960, ack 3625, win 1026, length 72
12:34:01.771748 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 960:1534, ack 3625, win 1026, length 574
12:34:01.917751 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 1534:1627, ack 3674, win 1026, length 93
12:34:02.105638 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [.], ack 3709, win 1025, length 0
12:34:04.584022 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 1627:1707, ack 3709, win 1025, length 80
12:34:04.584102 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 1707:2313, ack 3709, win 1025, length 606
12:34:04.726525 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 2313:2397, ack 3781, win 1025, length 84
12:34:04.727169 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 2397:2478, ack 3781, win 1025, length 81
12:34:04.727429 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 2478:3026, ack 3781, win 1025, length 548
12:34:04.869250 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [.], ack 3865, win 1025, length 0
12:34:04.884117 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 3026:3117, ack 3865, win 1025, length 91
12:34:19.071019 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [.], ack 3900, win 1025, length 0
12:34:19.270062 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 3117:3214, ack 3900, win 1025, length 97
12:34:19.271729 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [.], seq 3214:4646, ack 3900, win 1025, length 1432
12:34:19.271769 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 4646:5132, ack 3900, win 1025, length 486
12:34:19.413224 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [P.], seq 5132:5217, ack 3972, win 1024, length 85
12:34:19.468224 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [.], ack 4007, win 1024, length 0
12:34:19.607881 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [.], ack 4042, win 1024, length 0
12:34:21.367683 IP6 CLIENT.CLIENT_PORT > SERVER.https: Flags [R.], seq 5217, ack 4042, win 0, length 0
tcpdump on server (outbound)
$ sudo tcpdump -i eth0 dst 1.1.1.1
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C
0 packets captured
6 packets received by filter
0 packets dropped by kernel
// No outbound packets at all. Is caddy broken or connections to server failed?
sophauer commented
try to remove "h3" in alpn and protocols sections.]
and "protocol_min": "tls1.3", could cause some strange issues in my vps. and i had no idea why.removed it ,and it all worked
ArcCal commented
近一年多来，naive就没正常用过，新搭建的都总是活不过一小时。我是设置了最低tls1.3的，不知道是否与这个有关
sophauer commented
近一年多来，naive就没正常用过，新搭建的都总是活不过一小时。我是设置了最低tls1.3的，不知道是否与这个有关
好像跟h2有关,我在alpn配置里如果有h2,遇到的问题跟你一样,只留下http/1.1,就好了,已经存活好几个月了
ArcCal commented
好咧 我试试。不知道小火箭的fragment是什么原理。naive官方客户端连不上的节点，小火箭可以连得上，如果连不上开启fragment就又可以了。
Chilledheart commented
体感是TLS协议上出了问题而不是TCP，你可以把tcpdump输出成文件然后scp 到本地用wireshark打开，把TLS协议握手部分（hello）截图
具体操作是
tcpdump -i eth0 host CLIENT and port 443 -w naive.pcap
scp REMOTE:naive.pcap .
Chilledheart commented
把tcpdump命令里的dst 去掉，看看是否有正常的server hello
4 0.162484 192.168.2.22 → 1.2.3.4 TLSv1 583 Client Hello
5 0.325353 1.2.3.4 → 192.168.2.22 TCP 66 443 → 47402 [ACK] Seq=1 Ack=518 Win=43008 Len=0 TSval=3121284769 TSecr=4184841613
6 0.327746 1.2.3.4 → 192.168.2.22 TLSv1.3 1514 Server Hello, Change Cipher Spec, Application Data
gili-gili commented
把tcpdump命令里的dst 去掉，看看是否有正常的server hello
4 0.162484 192.168.2.22 → 1.2.3.4 TLSv1 583 Client Hello 5 0.325353 1.2.3.4 → 192.168.2.22 TCP 66 443 → 47402 [ACK] Seq=1 Ack=518 Win=43008 Len=0 TSval=3121284769 TSecr=4184841613 6 0.327746 1.2.3.4 → 192.168.2.22 TLSv1.3 1514 Server Hello, Change Cipher Spec, Application Data
yep
体感是TLS协议上出了问题而不是TCP，你可以把tcpdump输出成文件然后scp 到本地用wireshark打开，把TLS协议握手部分（hello）截图
具体操作是
tcpdump -i eth0 host CLIENT and port 443 -w naive.pcapscp REMOTE:naive.pcap .
is that a typical TCP RST attack
Client: VPS in Guangdong, China
gili-gili commented
Update: there are two messages
Good news, it has nothing to do with attackers. I ran naiveproxy on my setup even the vps itself and got the same result.
The bad one, obviously, naiveproxy, or caddy itself, is broken.
Not very sure about this, still investigating
gili-gili commented
I need help... It's beyond my ability.......
klzgrad commented
[0418/015445.268:INFO:naive_proxy_delegate.cc(137)] [https://example.com:443] negotiated padding type: None
This means your server isn't configured correctly and is not using the naiveproxy procotol.
gili-gili commented
Thanks for pointing out the problem sir.
I tested with minimal JSON configuration and got the same result.
Perhaps caddy upstream code changed something and caused naive to fail, or I built caddy with wrong commands so naive didn't replace the original code. The command I use is here, just as you provided in README file.
go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
~/go/bin/xcaddy build --with github.com/caddyserver/forwardproxy=github.com/klzgrad/forwardproxy@naive
klzgrad commented
You can start with a Caddyfile.
gili-gili commented
I think I found the problem:
{
"match": [
{
"host": ["example.com"]
}
],
"handle": ...naive config here...
}
The match block in route block will cause naive to fail. However the match block is required for a server running with multiple websites.
Don't use match together with the naive part config. I suggest to mention this in the documentation since json file is need for people with advanced usages.
Related issue: #620