tap_user usability tracking
fish4terrisa-MSDSM opened this issue · 33 comments
( Sorry for my poor english)
The QEMU has a feature to bind the guest`s port to the host\s port.Will RVVM support it later?It\s impossible to support x11 or SDL on android,so the vnc is the only choice.It`s important to make it possible to connect to the guest using the network port.
Yeah it will be there in future. The networking is WIP as you can see from README, my attention is pretty scrambled over multiple things.
Can we convert this issue into tap_user usability tracking issue?
P.S. Is there a way to draw a raw framebuffer on Android using native GUI APIs? I am going to add JNI support and then we can have native Android RVVM app without VNC shenanigans. A while ago I expected SDL to be usable on Android and only later realized it isn't...
Current tap_user issues/TODOs:
- No way to do 2-tuple lookup in TCP, because hashmap API is limited. This prevents us from doing 2-tuple reuse and therefore TCP port forwarding is broken.
No TCP buffering, it just expects the guest is able to receive all packets in time. Sometimes this results in dead connections,pacmanfailing to download packages, etc etc.- Inefficiend locking, it could be much more scalable and with less overhead
- Basic firewall/traffic limiting
Maybe it is worth to add VNC support into RVVM itself. There is such feature in QEMU (It acts like a VNC server), but I am unaware how complicated that might be (At least we have a nice networking lib).
Doing that via guest means is arguably not an ideal solution (We don't see the boot process, requires user intervention, what about non-Linux guests?), and we need a proper Android port anyways. So I guess JNI + android GUI is the best long term solution for this usecase.
Yeah it will be there in future. The networking is WIP as you can see from README, my attention is pretty scrambled over multiple things. Can we convert this issue into
tap_userusability tracking issue?P.S. Is there a way to draw a raw framebuffer on Android using native GUI APIs? I am going to add JNI support and then we can have native Android RVVM app without VNC shenanigans. A while ago I expected SDL to be usable on Android and only later realized it isn't...
Ok i will change the issue name.
P.S. I dont know much about the android api .(in fact,my project heavily depends on alpine-term.I dont found any android app can draw a raw fb,even the termux-x11 or xserver xsdl dont use it).Besides, adding gui support is not so important i think .Many people dont need the gui support on android,they may prefer terminal .And if the port forwarding and the gpu support is done ,the user can start a vnc by themselves.A unused vnc causes profermance decrease. If you want to build the own RVVM app ,editing on my project may be a good idea ,just delete the descriptions of archlinux and you will get one .(My project is still undone ,it will take me about 2 weeks to fully move from QEMU to RVVM .And if its done ,i will notice you in the discussions area)
Maybe it is worth to add VNC support into RVVM itself. There is such feature in QEMU (It acts like a VNC server), but I am unaware how complicated that might be (At least we have a nice networking lib).
Doing that via guest means is arguably not an ideal solution (We don't see the boot process, requires user intervention, what about non-Linux guests?), and we need a proper Android port anyways. So I guess JNI + android GUI is the best long term solution for this usecase.
Thats true ,I prefer the vnc method ,its also useful on other platforms.The Android GUI method is a bit complicated (termux-x11 might be a exmaple , it has developed for more than 3y but still have large amounts of bugs) ,but adding a vnc viewer in the app might be a good idea as the user can connect to the machine without using other apps.
A good vnc viewer project is avnc, its license is GPL3.You can insert some parts of it into the app.
Commit 833a16e solves all previous TCP traffic issues and running out of UDP ports.
It also includes groundwork for TCP NAT (host->vm connections) but the way to lookup socket via 2-tuple is still in the works.
Please report if you still have any TCP stability issues.
Figured there is an issue with keepalive ACKs not being properly handled. Fixed locally, will be upstreamed soon.
(This only matters if you have completely idle connections that live for hours)
TCP port forwarding works. However, there is an issue when connecting directly to 127.0.0.1 from the host, the guest gets a connection from 127.0.0.1 as well and gets confused, so the connection is dropped. I don't know how to solve that correctly yet.
Will be upstreamed soon.
A bug occured on NetBSD:
[root@archlinux ~]# echo $LANG
C.UTF-8
[root@archlinux ~]# curl www.google.com
WARN: net_poll_add() failed!
curl: (1) Received HTTP/0.9 when not allowed
[root@archlinux ~]# curl https://www.google.com
curl: (35) OpenSSL/3.0.8: error:0A0000C6:SSL routines::packet length too long
[root@archlinux ~]#
and when run curl www.google.com --http0.9,it will output the index.html of google,but the connection won`t close before timeout,so i think both the openssl error and the curl error is because that tcp connections cannot close accurately on NetBSD.The ping runs correctly ,so icmp have no problems.
When runs curl ip.sb --http0.9 to get the ip , it outputed
[root@archlinux ~]# curl ip.sb
curl: (1) Received HTTP/0.9 when not allowed
[root@archlinux ~]# curl ip.sb --http0.9
TTP/1.1 200 OK
Date: Tue, 09 May 2023 01:38:16 GMT
Content-Type: text/plain
Content-Length: 13
Connection: keep-alive
Cache-Control: no-cache
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=cgybpWVU5jI3iZjC8NPmm7lMwYhsUfoaNkW7zU5VFKGN%2BxbuCpdO9E1Z%2Fl%2BKK3Rz3gAVbsJx9Pm9q9B9LGOwd0Lrt4dE5hLwJK1y4pfyf3X3ut8XYFoV"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 7c4638d53bc13087-SEA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
205.166.94.4
The 'H' of "HTTP/1.1" disappeared ,so both the length of the packet and the head of the packets might be wrong(?)
Output of ifconfig
[root@archlinux ~]# ifconfig
enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.100 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::8c2e:4fff:fed9:45f prefixlen 64 scopeid 0x20<link>
ether 8e:2e:4f:d9:04:5f txqueuelen 1000 (Ethernet)
RX packets 45 bytes 10639 (10.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 79 bytes 6345 (6.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
The dhcp just worked well.I haven`t tested udp connections yet,and I also don`t know much about how to test it.
p.s. : now the VMA worked well on sdf.Great!
A bug occured on NetBSD:
[root@archlinux ~]# echo $LANG C.UTF-8 [root@archlinux ~]# curl www.google.com WARN: net_poll_add() failed! curl: (1) Received HTTP/0.9 when not allowed [root@archlinux ~]# curl https://www.google.com curl: (35) OpenSSL/3.0.8: error:0A0000C6:SSL routines::packet length too long [root@archlinux ~]#and when run
curl www.google.com --http0.9,it will output the index.html of google,but the connection wont close before timeout,so i think both the openssl error and the curl error is because that tcp connections cannot close accurately on NetBSD.The ping runs correctly ,so icmp have no problems. When runscurl ip.sb --http0.9` to get the ip , it outputed[root@archlinux ~]# curl ip.sb curl: (1) Received HTTP/0.9 when not allowed [root@archlinux ~]# curl ip.sb --http0.9 TTP/1.1 200 OK Date: Tue, 09 May 2023 01:38:16 GMT Content-Type: text/plain Content-Length: 13 Connection: keep-alive Cache-Control: no-cache CF-Cache-Status: DYNAMIC Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=cgybpWVU5jI3iZjC8NPmm7lMwYhsUfoaNkW7zU5VFKGN%2BxbuCpdO9E1Z%2Fl%2BKK3Rz3gAVbsJx9Pm9q9B9LGOwd0Lrt4dE5hLwJK1y4pfyf3X3ut8XYFoV"}],"group":"cf-nel","max_age":604800} NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} Server: cloudflare CF-RAY: 7c4638d53bc13087-SEA alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400 205.166.94.4The 'H' of "HTTP/1.1" disappeared ,so both the length of the packet and the head of the packets might be wrong(?) Output of
ifconfig[root@archlinux ~]# ifconfig enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.0.100 netmask 255.255.255.0 broadcast 192.168.0.255 inet6 fe80::8c2e:4fff:fed9:45f prefixlen 64 scopeid 0x20<link> ether 8e:2e:4f:d9:04:5f txqueuelen 1000 (Ethernet) RX packets 45 bytes 10639 (10.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 79 bytes 6345 (6.1 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0The dhcp just worked well.I haven
t tested udp connections yet,and I also dont know much about how to test it. p.s. : now the VMA worked well on sdf.Great!
I just tried curl --output ip.sb.out ip.sb --http0.9 -m 20,but the output is:
/root__curl --output ip.sb.out ip.sb --http0.9 -m 20 05/09/2023 09:47:39 AM
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 602 0 602 0 0 30 0 --:--:-- 0:00:20 --:--:-- 0
curl: (28) Operation timed out after 20018 milliseconds with 602 bytes received
/root__ls 05/09/2023 09:48:07 AM
_───_───────────_──────_───────_──────────_
│ # │ name │ type │ size │ modified │
├───┼───────────┼──────┼───────┼──────────┤
│ 0 │ ip.sb.out │ file │ 602 B │ now │
_───┴───────────┴──────┴───────┴──────────_
/root__cat ./ip.sb.out 05/09/2023 09:48:09 AM
TTP/1.1 200 OK
Date: Tue, 09 May 2023 01:47:50 GMT
Content-Type: text/plain
Content-Length: 13
Connection: keep-alive
Cache-Control: no-cache
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=%2B9y26TUxB6H%2FDgXjYX%2Fqi1JhdVY9i3aeAMhRmG6SPz8mgvaDPSuw8pk0cJ3tpiAhzive8aTSb3E7kP1BNUmkLCEFn9ZwlvFTcqNG%2BKEi511bq5ceN2Y8"}],"group":"cf-nel","max_age":604800}NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 7c4646d5be7608a1-SEA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
205.166.94.4
It seems that curl can`t detect the end of the connection.
The 'H' of "HTTP/1.1" disappeared
Ideally I'd need a full tcpdump during this curl invocation, thanks. Will look what it might be in the meantime.
WARN: net_poll_add() failed!
Oh, this is unexpected. Usually this indicates host resource exhaustion and only seen to happen on Windows with >1024 open connections. It is unexpected from kqueue to act like this, and this may also be the reason of connections not being closed.
Can you try commenting out this line and re-testing TCP?
Line 60 in 226330d
Weird... kqueue udata is not a void* in NetBSD???
The 'H' of "HTTP/1.1" disappeared
Ideally I'd need a full
tcpdumpduring this curl invocation, thanks. Will look what it might be in the meantime.
WARN: net_poll_add() failed!Oh, this is unexpected. Usually this indicates host resource exhaustion and only seen to happen on Windows with >1024 open connections. It is unexpected from
kqueueto act like this, and this may also be the reason of connections not being closed. Can you try commenting out this line and re-testing TCP?Line 60 in 226330d
Commented . It worked!!! Both bug is fixed.Great.Maybe a port of NetBSD can be added.
Yes...I got that warning when compling RVVM on sdf.
Yes...I got that warning when compling RVVM on sdf.
This basically means NetBSD violates kqueue API, and either needs workarounds or will have to use slower select implementation which is already used on Windows and Haiku as a safe generic implementation. Commenting out KQUEUE_NET_IMPL disables kqueue.
Fixed another (rare) kqueue bug, perhaps it hasn't bothered anyone yet but there it is
Weird warning in lynx browser. May be related to FIN/RST flags handling.
Now RVVM seems not support multi task downloading (I dont know the right way to describe that),will RVVM support it?
(As it showed , the other download task is rather slow except one)
Now RVVM seems not support multi task downloading
I am not sure at all that it's on RVVM side. I occasionally see it's balancing multiple downloads properly, sometimes not, and I have no idea what could be going wrong (I am just pumping data from a socket into ethernet frames).
It might be just that the download speed is peaking on one download, and other ones actually are received at that speed on the host.
However, running iperf in clean conditions over local network seems to distribute bandwidth evenly:
I will examine if, for example it accidentally prioritizes one socket over others.
You could also try to comment out this and re-test:
Line 54 in 5ab8066
Running with epoll on Illumos. Apparently it balances the downloads fine, and it still peaks at around 4.5 MB/s downlink - which makes me believe this is actually peak of the Arch RISC-V mirrors capabilities (per IP?).
So far I am almost convinced this is related environment (mirrors/host) rather than RVVM.
TCP port forwarding works. However, there is an issue when connecting directly to 127.0.0.1 from the host, the guest gets a connection from 127.0.0.1 as well and gets confused, so the connection is dropped. I don't know how to solve that correctly yet. Will be upstreamed soon.
Is this problem fixed now? And, there isn't an option matched with port forward, maybe an option like rvvm -hostfwd ${GUEST_PORT}:${HOST_PORT} can be added? That may enable me to mount the guest fs to the host(or a 9pfs like qemu might be a better choice, but currently RVVM didn't support it) And, although nearly nobody need udp port forwarding, -tcpfwd and -udpfwd might be a better choice?
I've almost finished a fix for localhost connections via port forwarding. Also working on distinguishing graceful FIN versus RST from outside. Will merge soon
Please test 89a01de, it should implement all the desired (so far) functionality (But no special arguments yet).
You can ssh into the guest with ssh root@localhost -p 2022.
Fixed a bunch of minor networking issues in staging branch.
@X547 In upcoming patch it will be possible to have a separate IP address for the guest, so basically a non-NAT scenario without tap_linux being used.
It relies on
- Being able to create multiple IP addresses on one adapter
- Being able to
bindtap_user sockets (both UDP/TCP) on that IP (Seenetworking.hAPI for details)
The guest will be accessing the network from that address now on. It doesn't need to be a physical adapter either, so you can seat the guest onto a veth/tap/whatever interface you want. Firewall rules also can work specifically on this guest.
It is then possible to disable emulated NAT in tap_user and directly map it's IP to the assigned one, and map any ports directly without port translation (Which should be configured separately)
IPv6 is also worked on, the remaining bits are ICMPv6 and proper guest addr assignment. I am not sure yet if above technique will work for IPv6 ranges, but at least I'm sure /128 addresses will work.
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether e4:a7:a0:e7:25:2a brd ff:ff:ff:ff:ff:ff
inet 192.168.0.203/24 brd 192.168.0.255 scope global dynamic noprefixroute wlan0
valid_lft 40563sec preferred_lft 40563sec
inet 192.168.0.37/24 scope global secondary wlan0
valid_lft forever preferred_lft forever
When running in unprivileged environment without additional configuration, IPv6 will be still usable for internet access, but will be NATed - there is pretty much no other option in unprivileged case
A separate IP address would be great, but is it possible to choose the ip address to bind with? And, it may cause ip address conflict if the user is not sure about whether the ip address is available in the network.
I think port forwarding is still useful, and options like -udpfwd and -tcpfwd would be great to see. Or, maybe a new option like -ip will help, and we may test whether the ip address is available before start the machine.
A separate IP address would be great, but is it possible to choose the ip address to bind with? And, it may cause ip address conflict if the user is not sure about whether the ip address is available in the network. I think port forwarding is still useful, and options like
-udpfwdand-tcpfwdwould be great to see. Or, maybe a new option like-ipwill help, and we may test whether the ip address is available before start the machine.
Basically there are going to be 2 options:
- Default: Guest simply has access to internet (can download stuff, etc) and is behind a virtual NAT (So it's not recognized as a separate network host from the outside point of view). You can use port forwarding to map some host ports to guest ports. Works on all OSes without special privileges.
- Interface passthrough: Guest sits on a separate address/interface so it is actually treated like a different machine in your LAN or even outside world in case of IPv6. Needs additional privileges, and unsure if it can be done on Windows. This is still WIP.
The latter option will be useful for more advanced usecases like firewalling the guest using host tools (iptables/netfilter/ipfw), accessing it as separate host without NAT, creating a virtual lan or putting it behind a VPN/proxy. It doesn't break simpler usecases (Which is to just get working internet everywhere)
Basically there are going to be 2 options:
- Default: Guest simply has access to internet (can download stuff, etc) and is behind a virtual NAT (So it's not recognized as a separate network host from the outside point of view). You can use port forwarding to map some host ports to guest ports. Works on all OSes without special privileges.
Great. That will make my current use unaffected. Port forwarding will really help.
- Interface passthrough: Guest sits on a separate address/interface so it is actually treated like a different machine in your LAN or even outside world in case of IPv6. Needs additional privileges, and unsure if it can be done on Windows. This is still WIP.
That seems attractive for me. Can you tell me which privileges (or, capabilities) in linux are needed?
The latter option will be useful for more advanced usecases like firewalling the guest using host tools (iptables/netfilter/ipfw), accessing it as separate host without NAT, creating a virtual lan or putting it behind a VPN/proxy. It doesn't break simpler usecases (Which is to just get working internet everywhere)
Hmm... It will also make it possible to use tcpdump to monitor the guest.
That seems attractive for me. Can you tell me which privileges (or, capabilities) in linux are needed?
CAP_NET_ADMIN for adding an address to interface; CAP_NET_BIND_SERVICE if you want to make guest ports below 1024 accessible without mapping them to a different port externally.
Systems without epoll or kqueue support won't be able to bind port ranges efficiently.
Implemented -portfwd CLI argument in c92727e.
Supported syntax:
-portfwd tcp/2022=22 - Forward TCP 0.0.0.0:2022 to guest port 22 (DHCP guest address is assumed)
-portfwd tcp/127.0.0.1:2022=22 - Forward TCP 127.0.0.1:2022 to guest port 22 (DHCP guest address is assumed)
-portfwd 25565=25565 - Forward both UDP & TCP 0.0.0.0:25565 to the same guest port
-portfwd udp/[::1]:6121=[fd9c:350d:af07::385]:6121 - Forward UDP [::1]:6121 to specific IPv6 address in guest network