Can't forward UDP to a subnet
Closed this issue · 15 comments
I want to forward UDP to a subnet, but I can't get connectivity.
I have been using this rules:
ip rule add fwmark 1088 table 100
ip route add local default dev eth2 table 100
iptables -t mangle -A OUTPUT -o eth2 -p udp -j MARK --set-mark 1088
iptables -t mangle -A PREROUTING -i eth2 -p udp -j TPROXY --on-ip 10.0.0.1 --on-port 10000 --tproxy-mark 1088
eth2 has 10.0.0.1 as IP.
In the socks5 proxy this log is showed, it tells the connection was done successfully:
# glider -verbose -listen socks5://127.0.0.1:9000
2021/10/08 21:16:03 group.go:186: [group] only 1 forwarder found, disable health checking
2021/10/08 21:16:03 server.go:38: [socks5] listening TCP on 127.0.0.1:9000
2021/10/08 21:16:03 server.go:107: [socks5] listening UDP on 127.0.0.1:9000
2021/10/08 21:16:13 server.go:150: [socks5u] 127.0.0.1:39434 <-> 1.1.1.1:53 via DIRECT
2021/10/08 21:17:16 server.go:150: [socks5u] 127.0.0.1:57056 <-> 1.1.1.1:53 via DIRECT
am I doing something wrong?
Looks like the socks5 server is mismatched. The UDP relay is not a standard implementation, It is UDP over TCP, allowing traffic to pass through the CDN. The socks5-tproxy needs to communicate with socks5-server.
I tested with other socks5 servers and the result is the same, and also hev-socks-server.
These are the logs:
root@localhost:/home/user# hev-socks5-server Temp3/New
[2021-10-08 23:10:19] [E] 0x7f793c001170 socks5 server req.ver 1
[2021-10-08 23:10:19] [E] 0x7f793c001170 socks5 server from sockaddr
[2021-10-08 23:10:24] [E] 0x7f793c001170 socks5 server req.ver 1
[2021-10-08 23:10:24] [E] 0x7f793c001170 socks5 server from sockaddr
[2021-10-08 23:10:29] [E] 0x7f793c001170 socks5 server req.ver 1
[2021-10-08 23:10:29] [E] 0x7f793c001170 socks5 server from sockaddr
root@localhost:/home/user# hev-socks5-tproxy Temp3/New2
[2021-10-08 23:10:07] [E] socks5 tproxy udp addr
[2021-10-08 23:10:19] [E] 0x55e056aedaf0 socks5 client auth.method 255
[2021-10-08 23:10:19] [E] 0x55e056aedaf0 socks5 session handshake
[2021-10-08 23:10:24] [E] 0x55e056aedaf0 socks5 client auth.method 255
[2021-10-08 23:10:24] [E] 0x55e056aedaf0 socks5 session handshake
[2021-10-08 23:10:29] [E] 0x55e056aedaf0 socks5 client auth.method 255
[2021-10-08 23:10:29] [E] 0x55e056aedaf0 socks5 session handshake
Just complementing, the connections come from 10.0.0.2 address, perhaps is this what causes this error, maybe hev-socks5-tproxy only works with 127.0.0.1 address?
May be. Try to set listening address to '::':1088 for tproxy and policy routing to lo dev:
ip rule add fwmark 1088 table 100
ip route add local default dev lo table 100
iptables -t mangle -A PREROUTING -p udp -j TPROXY --on-port 1088 --tproxy-mark 1088
iptables -t mangle -A OUTPUT -p udp -j MARK --set-mark 1088
/\ It doesn't work.
user@localhost:~$ hev-socks5-server Temp3/socks
[2021-10-09 20:10:01] [E] 0x7fb9c0001170 socks5 server req.ver 1
[2021-10-09 20:10:01] [2021-10-09 20:10:01] [E] 0x7fb9c0001170 socks5 server from sockaddr[E]
0x7fb9b8001170 socks5 server req.ver 1
[2021-10-09 20:10:01] [E] 0x7fb9b8001170 socks5 server from sockaddr
[2021-10-09 20:10:06] [E] 0x7fb9b8001170 socks5 server req.ver 1
[2021-10-09 20:10:06] [E] 0x7fb9b8001170 socks5 server from sockaddr
[2021-10-09 20:10:10] [E] 0x7fb9b8001170 socks5 server req.ver 1
[2021-10-09 20:10:10] [E] 0x7fb9b8001170 socks5 server from sockaddr
[2021-10-09 20:10:11] [E] 0x7fb9c0001170 socks5 server req.ver 1
[2021-10-09 20:10:11] [E] 0x7fb9c0001170 socks5 server from sockaddr
[2021-10-09 20:10:30] [E] 0x7fb9b8001170 socks5 server req.ver 1
[2021-10-09 20:10:30] [E] 0x7fb9b8001170 socks5 server from sockaddr
^Croot@localhost:/home/user# hev-socks5-tproxy Temp3/tproxy
[2021-10-09 20:09:58] [E] socks5 tproxy udp addr
[2021-10-09 20:10:01] [E] 0x558767ba8af0 socks5 client auth.method 255
[2021-10-09 20:10:01] [E] 0x558767ba8af0 socks5 session handshake
[2021-10-09 20:10:01] [E] 0x558767ba82b0 socks5 client auth.method 255
[2021-10-09 20:10:01] [E] 0x558767ba82b0 socks5 session handshake
[2021-10-09 20:10:06] [E] 0x558767ba82b0 socks5 client auth.method 255
[2021-10-09 20:10:06] [E] 0x558767ba82b0 socks5 session handshake
[2021-10-09 20:10:10] [E] 0x558767ba82b0 socks5 client auth.method 255
[2021-10-09 20:10:10] [E] 0x558767ba82b0 socks5 session handshake
[2021-10-09 20:10:11] [E] 0x558767ba82b0 socks5 client auth.method 255
[2021-10-09 20:10:11] [E] 0x558767ba82b0 socks5 session handshake
[2021-10-09 20:10:30] [E] 0x558767ba82b0 socks5 client auth.method 255
[2021-10-09 20:10:30] [E] 0x558767ba82b0 socks5 session handshake
user@localhost:~$ dig @1.1.1.1 g.co
; <<>> DiG 9.16.15-Debian <<>> @1.1.1.1 g.co
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
Check authentication on both sides.
Don't forgot to bypass socks5 server traffic to avoid infinite recursion if you run both on same host.
- Match by ip address
- Match by socks5 server process user id
- Match by cgroup
- ...
Sorry, I'm a little newbie.
How can I do that things you said?
Sorry, I'm a little newbie.
How can I do that things you said?
For normal case, the server is not run on local side, so if you only transparent proxy udp traffic, it is not need to config bypass rules.
{udp -> tproxy} -> (tcp) -> server
{}: local host
For test case, If you run server and tproxy on same host, like this:
{ udp1 -> tproxy -> (tcp) -> server -> udp2 }
that needs to bypass udp2, to avoid capture by tproxy again.
There is a simple way is match by process uid:
- Run server on special user. e.g. nobody (means network traffic from nobody will not proxy)
- Match by uid in iptables.
iptables -t mangle -A PREROUTING -p udp -j TPROXY --on-port 1088 --tproxy-mark 1088
iptables -t mangle -A OUTPUT -m owner --uid-owner nobody -j RETURN
iptables -t mangle -A OUTPUT -p udp -j MARK --set-mark 1088
I tested as you said.
hev-socks5-tproxy configuration:
socks5:
port: 9000
address: 127.0.0.1
tcp:
port: 1088
address: '::'
udp:
port: 1088
address: '::'
hev-socks5-server configuration (no authentication):
main:
# Worker threads
workers: 4
# Listen port
port: 9000
# Listen address (ipv4|ipv6)
listen-address: '::'
# Bind source address (ipv4|ipv6)
bind-address: '::'
Rules added:
ip rule add fwmark 1088 table 100
ip route add local default dev lo table 100
iptables -t mangle -A PREROUTING -p udp -j TPROXY --on-port 1088 --tproxy-mark 1088
iptables -t mangle -A OUTPUT -m owner --uid-owner debian-tor -j RETURN
iptables -t mangle -A OUTPUT -p udp -j MARK --set-mark 1088
hev-socks5-tproxy with logs:
root@localhost:/home/user# hev-socks5-tproxy Temp3/tproxy
[2021-10-10 04:31:28] [E] socks5 tproxy udp addr
[2021-10-10 04:31:39] [E] 0x5576983c7040 socks5 client connect
[2021-10-10 04:31:39] [E] 0x5576983c7040 socks5 client connect
[2021-10-10 04:31:39] [E] 0x5576983c7040 socks5 session connect
[2021-10-10 04:32:20] [E] 0x5576984022c0 socks5 client res.rep 1
[2021-10-10 04:32:20] [E] 0x5576984022c0 socks5 session handshake
[2021-10-10 04:33:20] [E] 0x557698402520 socks5 client read auth
[2021-10-10 04:33:20] [E] 0x557698402520 socks5 session handshake
[2021-10-10 04:34:06] [E] 0x5576983c70e0 socks5 client connect
[2021-10-10 04:34:06] [E] 0x5576983c70e0 socks5 client connect
[2021-10-10 04:34:06] [E] 0x5576983c70e0 socks5 session connect
hev-socks5-server running as the same user of iptables rules, with logs (an infinite loop of messages was showed):
root@localhost:/home/user# sudo -u debian-tor hev-socks5-server Temp3/tproxy
[2021-10-10 04:37:02] [2021-10-10 04:37:02] [E] [E] socks5 proxy acceptsocks5 proxy accept
[2021-10-10 04:37:02] [E] [2021-10-10 04:37:02] socks5 proxy accept[E]
socks5 proxy accept
[2021-10-10 04:37:02] [E] socks5 proxy accept
[2021-10-10 04:37:02] [E] socks5 proxy accept
[2021-10-10 04:37:02] [E] socks5 proxy accept
[2021-10-10 04:37:02] [E] socks5 proxy accept
[2021-10-10 04:37:02] [E] socks5 proxy accept
[2021-10-10 04:37:02] [E] socks5 proxy accept
[2021-10-10 04:37:02] [2021-10-10 04:37:02] [E] [E] socks5 proxy acceptsocks5 proxy accept
[2021-10-10 04:37:02] [E] socks5 proxy accept
[2021-10-10 04:37:02] [2021-10-10 04:37:02] [E] [E] socks5 proxy acceptsocks5 proxy accept
[2021-10-10 04:37:02]
[E] socks5 proxy accept
[2021-10-10 04:37:02] [E] socks5 proxy accept[2021-10-10 04:37:02]
[E] socks5 proxy accept[2021-10-10 04:37:02]
[E] socks5 proxy accept
[2021-10-10 04:37:02] [E] socks5 proxy accept
[2021-10-10 04:37:02] [2021-10-10 04:37:02] [E] [E] socks5 proxy acceptsocks5 proxy accept
As I see, the socks5 server received the connection of tproxy server, but it can't complete the UDP connection to outside:
root@localhost:/home/user# dig @1.1.1.1 t.co
; <<>> DiG 9.16.15-Debian <<>> @1.1.1.1 t.co
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
I captured the loopback queries in the same moment that a UDP query is made through tproxy server: packet_log.pcapng.zip
My kernel is a bit old, 5.4.20, I will made some tests in another computer with a newer kernel.
The tproxy server and socks5 server are working:
I think it's my iptables rules that are missing something that prevents the UDP packet for going through sub-net (eth2).
Can someone help me?
What is your network topology? And what to do?
I'm testing with a network namespace, I tested with a wifi hotspot and didn't worked too.
I use this commands to setup a network namespace with a virtual ethernet pair:
ip netns add nsx
ip li add vethx type veth peer name peerx netns nsx
ip li set vethx up
ip addr add 10.0.0.1/24 dev vethx
ip netns exec nsx ip li set lo up
ip netns exec nsx ip li set peerx up
ip netns exec nsx ip addr add 10.0.0.2/24 dev peerx
ip netns exec nsx ip route add default via 10.0.0.1 dev peerx
And the rules:
ip rule add fwmark 1088 table 100
ip route add local default dev vethx table 100
iptables -t mangle -A OUTPUT -o vethx -p udp -j MARK --set-mark 1088
iptables -t mangle -A PREROUTING -i vethx -p udp -j TPROXY --on-ip 10.0.0.1 --on-port 10000 --tproxy-mark 1088
sysctl -w net.ipv4.conf.vethx.forwarding=1
Everything is ok with tproxy server and socks5 server, I think now what is having problem is the iptables rules, maybe something is missing that prevents the traffic from tproxy server reaches the peer of virtual ethernet added.
If you wish, close this issue, there is nothing to do with tproxy and socks5 server.
I'm testing with a network namespace, I tested with a wifi hotspot and didn't worked too.
I use this commands to setup a network namespace with a virtual ethernet pair:
ip netns add nsx ip li add vethx type veth peer name peerx netns nsx ip li set vethx up ip addr add 10.0.0.1/24 dev vethx ip netns exec nsx ip li set lo up ip netns exec nsx ip li set peerx up ip netns exec nsx ip addr add 10.0.0.2/24 dev peerx ip netns exec nsx ip route add default via 10.0.0.1 dev peerxAnd the rules:
ip rule add fwmark 1088 table 100 ip route add local default dev vethx table 100 iptables -t mangle -A OUTPUT -o vethx -p udp -j MARK --set-mark 1088
If you don't want to proxy udp from gateway, remove this rule in OUTPUT chain. else you should add bypass rules to avoid the udp packets that from tproxy to namespace nsx are captured again.
and I tested with network namespace by your configs, It works fine while remove rules for OUTPUT.
iptables -t mangle -A PREROUTING -i vethx -p udp -j TPROXY --on-ip 10.0.0.1 --on-port 10000 --tproxy-mark 1088
sysctl -w net.ipv4.conf.vethx.forwarding=1
Everything is ok with tproxy server and socks5 server, I think now what is having problem is the iptables rules, maybe something is missing that prevents the traffic from tproxy server reaches the peer of virtual ethernet added. If you wish, close this issue, there is nothing to do with tproxy and socks5 server.
Thank you very much, I removed the rule in the OUTPUT chain and now it works!
root@localhost:/home/user# ip netns exec nsx dig @1.1.1.1 t.co
; <<>> DiG 9.16.15-Debian <<>> @1.1.1.1 t.co
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53960
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;t.co. IN A
;; ANSWER SECTION:
t.co. 9 IN A 104.244.42.5
;; Query time: 24 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Mon Oct 11 00:55:22 EDT 2021
;; MSG SIZE rcvd: 49
=)