microsoft/vscode

SSH/DevContainer Port Forwarding broken

max06 opened this issue · 17 comments

max06 commented

Does this issue occur when all extensions are disabled?: Yes/No

Version: 1.82.0-insider (system setup)
Commit: 3cd6f481266dcbd2ca2fcff43b4465d747c78e2f
Date: 2023-08-31T17:34:55.916Z
Electron: 25.7.0
ElectronBuildId: 23434598
Chromium: 114.0.5735.289
Node.js: 18.15.0
V8: 11.4.183.29-electron.0
OS: Windows_NT x64 10.0.22621

Issue similar to #190859 - this time it affects Devcontainers running on remote ssh hosts ⚠️

Client: Windows 11
Remote-Host: Ubuntu 22.04
Container: Debian 11
Repo for reproduction: https://github.com/gecio/gecio.github.io

Steps to Reproduce:

  1. Connect to ssh host
  2. Clone Repo
  3. Open in devcontainer when asked
  4. Launch Task "Serve" to start webserver
  5. Open http://localhost:4000 and see indefinite loading.

I can't test it without the remote ssh host :/

Shared-Log 

2023-09-01 00:51:07.658 [info] [SharedProcessTunnelService] Created tunnel 1: 127.0.0.1:64076 (local) to 127.0.0.1:7863 (remote).
2023-09-01 00:51:07.673 [info] [SharedProcessTunnelService] Created tunnel 2: 127.0.0.1:33761 (local) to 127.0.0.1:33761 (remote).
2023-09-01 00:51:07.833 [info] [SharedProcessTunnelService] Created tunnel 3: localhost:4000 (local) to localhost:4000 (remote).
2023-09-01 00:51:07.834 [info] [SharedProcessTunnelService] Created tunnel 4: localhost:35729 (local) to localhost:35729 (remote).
2023-09-01 00:51:09.216 [info] Getting Manifest... ms-vscode-remote.remote-containers
2023-09-01 00:51:09.313 [info] Installing extension: ms-vscode-remote.remote-containers
2023-09-01 00:51:10.103 [info] Extension signature is verified: ms-vscode-remote.remote-containers
2023-09-01 00:51:10.426 [info] Extracted extension to file:///c%3A/Users/max06/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.308.0: ms-vscode-remote.remote-containers
2023-09-01 00:51:10.432 [info] Renamed to c:\Users\max06\.vscode-insiders\extensions\ms-vscode-remote.remote-containers-0.308.0
2023-09-01 00:51:10.439 [info] Extracting extension completed. ms-vscode-remote.remote-containers
2023-09-01 00:51:10.463 [info] Extension installed successfully: ms-vscode-remote.remote-containers
2023-09-01 00:51:10.469 [info] Marked extension as uninstalled ms-vscode-remote.remote-containers-0.307.0
2023-09-01 00:51:12.628 [info] [SharedProcessTunnelService] Created tunnel 5: 127.0.0.1:42025 (local) to 127.0.0.1:42025 (remote).
2023-09-01 00:51:30.294 [info] Creating a socket (renderer-Tunnel-f2cc70c7-d306-4708-b542-3bfdd2cb513f)...
2023-09-01 00:51:30.416 [info] Creating a socket (renderer-Tunnel-f2cc70c7-d306-4708-b542-3bfdd2cb513f) was successful after 123 ms.
2023-09-01 00:51:30.957 [info] Creating a socket (renderer-Tunnel-57cbefc6-cb29-42ef-89ac-2ff5873ad23a)...
2023-09-01 00:51:31.049 [info] Creating a socket (renderer-Tunnel-57cbefc6-cb29-42ef-89ac-2ff5873ad23a) was successful after 93 ms.
2023-09-01 00:51:31.223 [info] Creating a socket (renderer-Tunnel-d3fcfdea-adbf-44f9-a6da-2d1089f1c7fd)...
2023-09-01 00:51:31.311 [info] Creating a socket (renderer-Tunnel-d3fcfdea-adbf-44f9-a6da-2d1089f1c7fd) was successful after 90 ms.
2023-09-01 00:56:31.298 [info] Creating a socket (renderer-Tunnel-ec10ac1a-bbec-46d3-bb84-2667f176bf2f)...
2023-09-01 00:56:31.400 [info] Creating a socket (renderer-Tunnel-ec10ac1a-bbec-46d3-bb84-2667f176bf2f) was successful after 101 ms.
2023-09-01 00:58:16.519 [info] Creating a socket (renderer-Tunnel-bebfb153-c346-414c-bdcf-e567fb8e8bf6)...
2023-09-01 00:58:16.623 [info] Creating a socket (renderer-Tunnel-bebfb153-c346-414c-bdcf-e567fb8e8bf6) was successful after 104 ms.
2023-09-01 00:58:16.783 [info] Creating a socket (renderer-Tunnel-54b3ca0c-f89e-4827-bd48-d42869c6094f)...
2023-09-01 00:58:16.891 [info] Creating a socket (renderer-Tunnel-54b3ca0c-f89e-4827-bd48-d42869c6094f) was successful after 109 ms.

Dev console doesn't contain anything related.

The extension host shows:

2023-09-01 00:53:21.861 [error] Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
2023-09-01 00:53:21.861 [error] Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
2023-09-01 00:53:21.985 [error] Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
2023-09-01 00:53:22.031 [error] Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
2023-09-01 00:53:22.451 [error] Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)

Although I'm not sure if that's related.

The server output is interesting:

2023-08-31 23:03:58.609 [error] Error: connect ECONNREFUSED ::1:4000
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
2023-08-31 23:03:58.870 [error] Error: connect ECONNREFUSED ::1:4000
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)

This is directly related - it happens right/shortly after opening the url in the browser.

connor4312 commented

Insiders has upgraded to Node 18 which now prefers ipv6 in dns lookups for localhost. I wonder if your service listens only on ipv4 localhost. When connecting to a tunnel we ask for localhost, though this might be configurable (@alexr00?)

https://github.com/microsoft/vscode/blob/main/src/vs/platform/tunnel/node/tunnelService.ts#L116C2-L116C2

On the server we could do what I did in microsoft/dev-tunnels#292 and try to connect both to ipv4 and ipv6 localhost for any forwarded connection asking for localhost, and use whatever is accepted first.

max06 commented

Using 127.0.0.1 has no effect...

Worth mentioning: My remote host and the docker network are both ipv6 enabled (dual stack)

connor4312 commented

I corrected my comment, that would not have worked.

Do you know if the server running in the repo bound to the ipv6 or ipv4 localhost?

max06 commented 
 $  ss -tulpn | grep 4000                                                                                                                                                       
tcp   LISTEN 0      4096       127.0.0.1:4000       0.0.0.0:*    users:(("bundle",pid=48387,fd=11))
connor4312 commented

Ah, okay. That seems to be what is happening. I imagine this may be a popular issue with our next release. Will discuss whether we want to put in a patch at this point.

max06 commented

If you pull the regular release to node 18 - that would be great. I only work in devcontainers... 😅
If not... I can move those containers that use port forwarding back to the regular version.

connor4312 commented

Local/dev verification steps:

  1. Have two machines on an ipv6-enabled network, and 'remote' and 'local'
  2. Modify extensions/vscode-test-resolver/src/extension.ts to return { connectionToken: 'test', host: '<ip or host>', port: 9888 } from the resolve method on the test resolver, and comment out its tunnelFactory
  3. Run ./scripts/code-server.sh --host=0.0.0.0 --connectionToken=test on the 'remote' machine
  4. On the 'local' machine, run OSS dev and run the command "Connect to TestResolver in Current Window"
  5. Forward port 3000 on the local machine
  6. On the remote machine, run npx serve -l "tcp://[::1]:3000". Verify you can hit it on the local machine. This should work in main.
  7. On the remote machine, run npx serve -l tcp://127.0.0.1:3000. Verify you can hit it on the local machine. This will fail 🐛
connor4312 commented

Labelling as candidate for discussion

chrmarti commented

7. On the remote machine, run npx serve -l tcp://127.0.0.1:3000. Verify you can hit it on the local machine. This will fail 🐛

@deepak1556 This sounds like the extension host uses ipv6 to connect to localhost. Wasn't --dns-result-order=ipv4first supposed to keep the Node 16 behavior of preferring ipv4?

deepak1556 commented

Yes the extension host prefers the legacy result order of ipv4 first, it is configured for both local and remote extension hosts

vscode/src/vs/server/node/extensionHostConnection.ts

Line 259 in 5f7b620

opts.execArgv.unshift('--dns-result-order=ipv4first');

vscode/src/vs/workbench/services/extensions/electron-sandbox/localProcessExtensionHost.ts

Line 252 in 5f7b620

opts.execArgv.unshift('--dns-result-order=ipv4first');

deepak1556 commented

I am not familiar with port forwarding code, but the change here #191950 is addressing the connection to the server. I have not changed the behavior for server main process, basically it will use the newer behavior of whatever the OS prefers.

Should the server also be changed to use the legacy result order ?

chrmarti commented

Not entirely sure, but it seems like it would make sense to keep the user-visible behavior the same and avoid surfacing the change in Node.js.

On the other hand, users will gradually upgrade to Node.js 18 and since server.listen also changes its behavior, the ports the forwarding code is connecting to will change interfaces. So maybe the autoSelectFamily flag you suggest or @connor4312's probing of both interfaces when connecting to localhost might be more future proof.

alexdima commented

The code that sets up a tunnel executes in the server process, where it tries to connect to localhost, so that is why the extension host flag doesn't have any effect. I also like the idea of using node's builtin autoSelectFamily rather than us reimplementing it.

  • 👍1
alexdima commented

I was able to reproduce via SSH to an Ubuntu 22 installation only after configuring /etc/hosts to add localhost to ::1, in the following way:

image

alexdima commented

The repro steps which worked for me:

  • install ubuntu 22.04.3 in Parallels
  • change /etc/hosts and add localhost to ::1 e.g.
127.0.0.1 localhost
127.0.1.1 ubuntu2204

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
  • connect via SSH
  • create a js file
  • npm install yaserver
const yaserver = require('yaserver');
const http = require('http');

const PORT = 3002;

yaserver.createServer({
	rootDir: __dirname
}).then((staticServer) => {
	const server = http.createServer((request, response) => {
		return staticServer.handle(request, response);
	});
	server.listen(PORT, '127.0.0.1', () => {
	// server.listen(PORT, '::1', () => {
	});
});
  • launch the file via the node installed by the server e.g. /home/alex/.vscode-server-insiders/cli/servers/Insiders-dd112ec0243c4b42bb3106e64df1688e242a6559/server/node server.js
  • add port forwarding for port 3002
  • open localhost:3002 in the local machine
max06 commented

I just want to raise the question if makes sense to build something complicated for detecting the address family now - not sure how often ipv6 is used for localhost binding?

I'd be perfectly fine if it always goes for ipv4 for the time being.

Edit: ... or I take the fix that just got finished while writing this comment 😂

max06 commented

Verified in latest insider, works.

Commit: f1302be1e67e3af5fbeb8bbb2ea784de7bc96150
Date: 2023-09-01T11:08:40.414Z

Thank you very much for saving me!

For documentation:

2023-09-01 12:31:17.112 [error] AggregateError
    at internalConnectMultiple (node:net:1081:18)
    at afterConnectMultiple (node:net:1532:5)

does now show up in the server output when opening the port. Doesn't seem to prevent it from working though.