Localhost relay does not support dual-mode sockets
justinbarclay opened this issue · 44 comments
Your Windows build number: (Type ver at a Windows Command Prompt)
Microsoft Windows [Version 10.0.19551.1005]
What you're doing and what's happening:
I am trying to start-up a webserver and have it listen on a specific port. Then when I try to reach this website using a service like localtest.me it says it's unreachable, this is specifically happening when I try to run shadow-cljs on WSL2. If I run it on a WSL distro it works fine. After some talking with the author of that package I was able to boil down a reproducible issue to:
- In a fresh instance of Ubuntu with WSL2 enabled run
nc :: 8080 -l - Try to connect to nc, on Windows with any browser, at the address 127.0.0.1:8080.
What's wrong / what should be happening instead:
When opening up a server and telling it to listen on '::' and port 8080, the server becomes unreachable on the Windows side when trying to access it through a browser on 127.0.0.1:8080. However, the server is reachable through the browser when accessing it through localhost:8080. This workflow used to work on WSL1. I have tested this on a multitude of ports and it never works.
Additionally, this works fine if I use nc 0.0.0.0 8080 -l instead. It is reachable by browser on 127.0.0.1 and localhost.
Try to connect to nc, on Windows with any browser, at the address 127.0.0.1:8080.
Right; you aren't listening on ipv4 127.0.0.1 in your repro.
If you try to connect to 127.0.0.1:8080 on the linux side, the Linux kernel routes those (different) protocols automatically. The Windows->WSL ipv4 127.0.0.1 tunnel, notsomuch.
on Windows with any browser
No; MS Edge and Chrome treat the string localhost differently (Edge favors resolving localhost to ipv4 over ipv6 AFAICT). The client (a browser or otherwise) matters. Your browser tried connecting on ipv6, because that's how localhost resolved.
Ref ongoing #4353 et al. Work-around for the time being: Use the real ipv4 subnet to listen. That can be ever-popular 0.0.0.0, or an ipv4 subnet that is routable from Windows. For example, listen on 127.0.0.1 or (for me atm) 192.168.181.47 or 192.168.181.0. Connect using an ip4 address (contrast an ipv6 address).
I wasn't sure if it was the same root cause as #4353, so I figured it was worth posting. Thank you for the response and work arounds.
Ran into this same issue trying to run an express server in wsl2. If you're using hostnames from your hosts file, a quick workaround is to just change them to point to your local IPv6 address instead of the v4 one, so 127.0.0.1 mysite.dev becomes ::1 mysite.dev.
Tag needs-investigation in the sense this needs a ruling on whether the current (circa 19640) behavior of the magic WSL localhost tunnel is by-design (by fiat), or should route ipv4 127.0.0.1 to ipv6 ::1 automatically like it does on Linux. I suspect but can't prove there is an IETF RFC that defines the behavior (because there's an IETF RFC for everything) but I didn't look it up.
The OP up top is sound, although you'll be better off using win32 curl.exe 127.0.0.1 to avoid ambiguity 'Edge'. New Edge (and Chrome) will look like there is no problem because it resolves localhost to ::1.
I just spent some time looking into issues I think are related to this. I have a NodeJS project in WSL2 opening a TCP/4000 server. Inside the WSL Bash I see it correctly binding:
$ netstat -apn | grep 4000
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::4000 :::* LISTEN 1771/node
Running netstat -a -p -n in cmd.exe however paints a slightly different picture:
TCP [::1]:4000 [::]:0 LISTENING 9140
Somehow the global binding inside wsl.exe translates to a ::1 binding in wslhost.exe (PID 9140).
Even more curious, when I use Docker Desktop WSL2 to launch an Nginx container with -p 80:80 it looks like this:
TCP [::]:80 [::]:0 LISTENING 23396
TCP [::1]:80 [::]:0 LISTENING 9140
So for some reason wslhost.exe is only providing the loopback bindings, and com.docker.backend.exe is forwarding it to the network.
Why does wslhost constrain the global binding to loopback itself?
We got hit by this on firebase/firebase-tools-ui#332 -- it turns all our Java programs (Firebase Emulators) experiences this issue since Java prefers to open up an IPv6 socket by default, under the assumption that it will serve both IPv4 and IPv6. (It does that even if asked to listen on 127.0.0.1, an IPv4 address.)
java -Djava.net.preferIPv4Stack=true mitigates this problem (and netstat -nl shows tcp instead of tcp6), but that prevents listening to both IPv4 and IPv6 at the same time.
@therealkenc Would it be reasonable for WSL2 to route ipv4 127.0.0.1 to ipv6 ::1 automatically like Linux? There may or may not be an IETF RFC, but a lot of programs like Java is already built under the assumptions and bridging that gap would help compatibility.
Would it be reasonable for WSL2 to route ipv4 127.0.0.1 to ipv6 ::1 automatically like Linux?
Yes. RFC 6052 (...or something)
I had this problem using Apache 2 and ended up having to change my Listen 93 directive to Listen 0.0.0.0:93
then it started working. This is because it uses ipv6 by default and there are known issues as of now.
If the WSL2 subsystem does not forward IPv6 traffic, why have IPv6 enabled? I have this in my bashrc to turn it off until we have a working IPv6 stack. I'd prefer that IPv6 work! Windows should really request an IPv6 subnet and apply that to the vEthernet network so that all hypervisors and WSL2 instances have working IPv6 addresses. It should then forward IPv6 connections. Alas, we're not there yet. Here's a simplified version of my workaround to disable IPv6. The sysctl instances are the key.
if grep -q "microsoft" /proc/version &>/dev/null; then
# WSL2
export DISPLAY="$(ip route|awk '/^default/{print $3}'):0.0"
export PULSE_SERVER="${PULSE_SERVER:-tcp:$(ip route|awk '/^default/{print $3}')}"
grep -q 1 /proc/sys/net/ipv6/conf/all/disable_ipv6 || sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
grep -q 1 /proc/sys/net/ipv6/conf/default/disable_ipv6 || sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
pgrep -xo rsyslogd >/dev/null || sudo service rsyslog start
pgrep -xo sshd >/dev/null || sudo service ssh start
pgrep -xo dbus-daemon >/dev/null || sudo service dbus start
pgrep -xo mysqld >/dev/null || sudo service mysql start
pgrep -xo php7.4-fpm >/dev/null || sudo service php7.4-fpm start
pgrep -xo apache2 >/dev/null || sudo service apache2 start
fi
Now when services listen, they default to listening on IPv4 instead of IPv6. This works for apache, mysql, java apps, but did not work for sshd in my instance.
I'm running OpenSSH under windows which gets me in to a powershell prompt where I start wsl.exe to get into WSL2. This means I don't need or want port 22 forwarded through WSL2 anyway.
If you want your WSL2 sshd to answer, you'll need to at least add ListenAddress 0.0.0.0 to /etc/ssh/sshd_config or wherever your sshd_config lives. This still by default only forwards the WSL2 port 22 to localhost and does not make it available remotely. Not sure how to work around that. Note: this requires you manually restarting WSL2 as it does not start on boot. Using the Windows sshd instance allows you to remotely start wsl.exe after connecting with ssh.
It appears resolving this issue would improve my use case as well. Our shared DNS defines a wildcard domain that points to localhost, such as *.local.domain.com. This allows devs to define any local app they want listening on localhost at those domains, app1.local.domain.com and app2.local.domain.com.
This is convenient as it avoids having to update hosts files all the time, but with this issue, it appears the only way to work around it currently is by actually creating host file entries for items, even if they already resolve to 127.0.0.1.
Has anyone tested disabling ipv6 as I posted above? Basically:
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
I don't know why the default WSL kernel has IPv6 enabled if WSL can't route IPv6.
Disabling ipv6 on Ethernet adapters (from windows) worked for me.
Disabling ipv6 on Ethernet adapters (from windows) worked for me.
That means breaking windows IPv6 connectivity, right? I'd prefer to keep it.
Or did you only disable IPv6 on the Hypervisor Ethernet?
Disabling ipv6 on Ethernet adapters (from windows) worked for me.
That means breaking windows IPv6 connectivity, right? I'd prefer to keep it.
Or did you only disable IPv6 on the Hypervisor Ethernet?
I disabled them all as I was not using them, but you can try only the WSL adapter.
I don't know why the default WSL kernel has IPv6 enabled if WSL can't route IPv6.
This isn't really true. It routes IPv6 fine, it just doesn't also bind to IPv4 when you bind only to an IPv6 address. If I start a server using IPv6, like python3 -m http.server --bind ::, then I can access it fine on the Windows side at http://[::1]:8000/, I just can't access it at http://127.0.0.1:8000/. The same is true the other way around, binding to 0.0.0.0 is only accessible via IPv4 on the Windows side.
This behavior isn't really wrong, it's just not the default behavior you see in most OSs. The only weird thing is that on the Linux side, it does operate in dual stack mode, so you'll run into issues if you try to explicitly bind to both IPv4 and IPv6 in your service. You can get around this by disabling dual-stack mode when you bind to IPv6. For a node.js server, that could look something like this:
const http = require("http");
const reqHandler = (req, res) => res.end("Hello World");
http.createServer(reqHandler).listen({ port: 8000, host: "::", ipv6Only: true });
http.createServer(reqHandler).listen({ port: 8000, host: "0.0.0.0" });With a setup like that, you will get a response on the Windows side using both http://127.0.0.1:8000/ and http://[::1]:8000/.
Obviously this isn't ideal, it would be nice if WSL forwarded both IPv4 & IPv6 when bound to an IPv6 address, especially since it works properly from the Linux side. But I wouldn't go as far as to say "WSL can't route IPv6".
Have you gotten IPv6 networking from the WSL2 side? What I meant is that the WSL2 image does not get a routable IPv6 address when it boots up. Neither outbound nor inbound IPv6 works for me. Is there a way of assigning IPv6 space to WSL2? ideally I'd like the windows host side to request an network block and assign IPv6 routes for each WSL instance out of that block, then route to the nodes. This would mean IPv6 from host to WSL2 and back works as well as WSL2 to and from external IPv6 hosts.
I reinstalled my debian and Ubuntu wsl2 distro and now I do have the same problem with Ruby on Rails.
geert@HTC0273:~/works/vfl$ bin/rails server
=> Booting Puma
=> Rails 6.1.3 application starting in development
=> Run `bin/rails server --help` for more startup options
Puma starting in single mode...
* Puma version: 5.2.2 (ruby 3.0.0-p0) ("Fettisdagsbulle")
* Min threads: 5
* Max threads: 5
* Environment: development
* PID: 24002
* Listening on http://127.0.0.1:3000
Use Ctrl-C to stop
I am not able to connect through my Chrome browser on the win10 host.
Any idea of a workaround for the time being?
Listening on http://127.0.0.1:3000
Get the server (by whatever means) to listen on 0.0.0.0:3000. If that doesn't help, try WSL's IPv4 address instead of localhost in the Windows-side browser.
- Listening on http://127.0.0.1:3000
I am not able to connect through my Chrome browser on the win10 host.Any idea of a workaround for the time being?
Try the sysctl settings I posted above to disable ipv6 inside the WSL2 instance.
#4851 (comment)
Disabling IPV6 didn't solve the problem for me. I am having the same issue using WSL2 with Ubuntu 20.04 and a web server and PHP dev configuration using ddev. I cannot access the server with the ddev describe URL, only with the IP address.
Does anyone have full IPv6 connectivity working inside WSL2 ? My Win 10 host has IPv6, but WSL2 sessions do not.
As far as I know, IPv6 is explicitly not supported in WSL 2, see https://docs.microsoft.com/de-de/windows/wsl/compare-versions#ipv6-access
@timriker I tried your sysctl method but it seems not working.
I tried to start a node server without specifying the host, which makes node.js bind it to :: because it finds ipv6 is enabled.
This server therefore cannot be accessed via host.docker.internal within a docker container started by Docker Desktop. The host.docker.internal works only if the server listens on ipv4.
@wizcas true, with the sysctl changes, many apps don't listen on ipv6, but node does.
The only work around I've found for node it to add an explicit ipv4 listen. ie:
var server = app.listen(port, "0.0.0.0", function () {
This is unfortunate because in a deployed environment I do want it to listen on ipv6.
@timriker
Yes that would be a workaround, but similar to your situation, I am not allowed to change the listen()'s parameter as well. 🙃
I guess I'll have to use Linux Docker engine instead of Docker Desktop until there's a fix from the WSL team... The good news is I don't really need to use Windows docker container.
I've got a similar problem:
WSL 2 running a node server listening on [::]:3000 (node listens only to ipv6)
Trying to connect via chrome to http://127.0.0.1:3000 wasn't working.
Then following the article shared by @stevenobird I tried:
netsh interface portproxy add v4tov4 listenport=3000 listenaddress=0.0.0.0 connectport=3000 connectaddress=<WSL IP>
And http://127.0.0.1:3000 worked
I had a problem with Golang HTTP server listening IPv6 :::3010 within WSL2 Ubuntu machine.
I can't reach it as 127.0.0.1:3010 from my Windows 10 and I've solved the issue redirecting IPv4 address to IPv6:
netsh interface portproxy add v4tov6 listenaddress=127.0.0.1 listenport=3010 connectaddress=::1 connectport=3010
CurrPorts software was really helpful to look into things on Windows.
And that command helped me to understand the Ubuntu situation:
sudo netstat -tulpn | grep LISTEN
Workaround, run in your wsl terminal:
socat tcp-l:8080,fork,reuseaddr tcp:127.0.0.1:9090
Where 8080 is the port you are trying to access via your browser, localhost:8080 and 9090 the service that is listening in your wsl on port 9090
Depending on the IP protocol your app is listening on, use ::1 or 127.0.0.1 in your windows host's file. You can check it by running sudo netstat -plunt | grep $appPort on the wsl machine.
Please provide transparent dual-stack routing to WSL ⭐
This came up recently here too. Build:
Version 10.0.19042 Build 19042
This continues to be an issue, and the only consistent solution I've found so far is this: https://abdus.dev/posts/fixing-wsl2-localhost-access-issue/
It's a little involved and requires you to use a differently named host than localhost, but it works. Essentially this sets up a task to be triggered when wsl starts which gets the dynamic IP address of the wsl2 instance and then adds it to your windows host file under the name wsl.
So let's say you have an application you normally access in wsl2 by going to localhost:8080 in your browser. After setting up this script you would go to wsl:8080.
You'd need to tweak any services that try to hit the thing running in the wsl2 instance to use this new wsl endpoint instead of localhost.
Tuve un problema con el servidor Golang HTTP escuchando IPv6 :::3010 dentro de la máquina WSL2 Ubuntu.
No puedo alcanzarlo como 127.0.0.1:3010 desde mi Windows 10 y resolví el problema al redirigir la dirección IPv4 a IPv6:
netsh interface portproxy add v4tov6 listenaddress=127.0.0.1 listenport=3010 connectaddress=::1 connectport=3010El software CurrPorts fue realmente útil para investigar cosas en Windows.
Y ese comando me ayudó a entender la situación de Ubuntu:
sudo netstat -tulpn | grep LISTEN
it worked for me
I believe you can get IPv6 with a public address if you advertise a prefix on the virtual WSL network adapter from within Windows. Then you'd have to instruct your router to route that prefix to your Windows host. While advertising can be set up with netsh, I am not aware that Windows can request a prefix from the upstream router. So manual setup of the routing table is required. Since many people are given dynamic prefixes, this is a bit cumbersome.
Reading the original bug filed I'm going to close this as by design.
ipv6 and ipv4 bindings are separate. (Except maybe for RFC 4291)
The original request tries to listen on :: 8080 which WORKS.
They they try to connect to :: 8080 from windows, which does NOT work, but should.
If they try to connect to localhost:8080 this WORKS, but is unexpected.
This still sounds like a bug to me. If the windows Hyper-V it NATting ipv4 traffic and routing 127.0.0.1 to WSL listening services, it should also NAT ipv6 and route ::1 traffic to WSL listening services.
fc00::/7 address space is set aside as unique local addresses. IMHO, WSL should use some of this space through the Hyper-V network adapter just as it uses 172.16/12
https://en.wikipedia.org/wiki/Unique_local_address
Developing and supporting IPv6 capable software under WSL is impossible currently as there is no IPv6 support under WSL. Please fix.
Concerning the original bug as written. Binding on :: 8080 (ipv6 wildcard) on the guest. They describe being unable to connect via 127.0.0.1:8080 (ipv4) on the host using the relay. This is expected. They then describe being able to connect via localhost:8080 in the browser on the host. This relies on implementation specific behavior in the browser on the host where it can connect to 127.0.01 or ::1 depending on what it feels like.
Note: Running Hyper-V Manager as Administrator, and then in Virtual Switch Manager edit the WSL switch and connect it to "External Network" and an ethernet adapter on the host, gets you ipv4 and ipv6 connectivity if that exists on the host network.
My ideal would be to have dual adapters inside WSL2. One that is NAT connected to the host, with RFC1918 addresses for IPv4 and fc00::/7 NATted ipv6 space. Then, if we're on Ethernet, an additional interface that is bridged to the host ethernet and gets it's own, perhaps public, IPv4 and IPv6 addresses from that network.
Doing this on WiFi has the issue that adding an additional MAC won't work in many cases, but it does work well on ethernet.
Two valid methods of getting WSL2 online with IPv6 as well as existing IPv4. No kernel change required, just setting up Hyper-V networks. I don't know that Hyper-V supports NAT on ipv6. I've not gotten that working, but it does support bridging with IPv4 and IPv6.
@timriker IMHO the best would be if the host could request a prefix via DHCPv6-PD and then route this to the WSL2 instance.
@timriker IMHO the best would be if the host could request a prefix via DHCPv6-PD and then route this to the WSL2 instance.
Yes, if the host can get a prefix, then use that. If it cannot, then fall back to NAT on fc00::/7
I'm re-opening this to keep track of the lack of support for dual-mode sockets in the localhost relay as indicated by the original bug. Any other issues should go to separate bugs.
What worked for me was editing my .wslconfig file and commenting out the following:
localhostforwarding=truenetworkingMode=mirroredhostAddressLoopback=true
Then running the following from PowerShell
wsl --shutdown
Here is my full configuration
# Settings apply across all Linux distros running on WSL 2
[wsl2]
# (CAUSED ISSUE) Turn on default connection to bind WSL 2 localhost to Windows localhost
# localhostforwarding=true
# (CAUSED ISSUE) Switch networking mode to mirrored
#networkingMode=mirrored
#hostAddressLoopback=true
# Disables nested virtualization
nestedVirtualization=false
# Turns on output console showing contents of dmesg when opening a WSL 2 distro for debugging
debugConsole=true
# Enable experimental features
[experimental]
sparseVhd=trueHi. Can you please collect networking logs by following the instructions below?
https://github.com/microsoft/WSL/blob/master/CONTRIBUTING.md#collect-wsl-logs-for-networking-issues
Hello folks. We don't need logs for this bug. Sorry for the confusion! We are able to reproduce it and know the source of the issue. It's just a question of priorities.