ekzhang/sshx

nginx: stream error received: unspecific protocol error detected

shumvgolove opened this issue · 7 comments

shumvgolove commented

Hey 👋 and thanks for the great project!

After several attempts to figure out how to make it work with nginx reverse-proxy, I've created half-working solution. I am providing below steps to reproduce the issue and ultimately figure out what is the problem.

sshx-server setup

  1. Create and enter temp folder:
mkdir /tmp/reproduce-issue && cd /tmp/reproduce-issue
  1. Clone and enter the sshx repository:
git clone https://github.com/ekzhang/sshx.git && cd sshx
  1. Build frontend:
npm ci && npm run build
  1. Build backend:
cargo build --release --bin sshx-server
  1. Launch sshx (which will also serve frontend).:
mv target/release/sshx-server . && ./sshx-server --secret dev-secret --listen :: --override-origin https://localhost:8080

nginx setup

  1. Create and enter nginx folder:
mkdir /tmp/reproduce-issue/nginx && cd /tmp/reproduce-issue/nginx
  1. Create nginx.conf with the following content:
# Run nginx using:
#     nginx -p $PWD -e stderr -c nginx.conf

daemon off;  # run in foreground

events {}

pid nginx.pid;

http {
    access_log /dev/stdout;

    # Locally generated SSL certificates from mkcert
    ssl_certificate ./localhost+1.pem;
    ssl_certificate_key ./localhost+1-key.pem;
    ssl_protocols	TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

    # Directories nginx needs configured to start up.
    client_body_temp_path .;
    proxy_temp_path .;
    fastcgi_temp_path .;
    uwsgi_temp_path .;
    scgi_temp_path .;
    
    ###################
    client_body_timeout 5;  # <------ makes sshx client spitting error log every 5 seconds
    ###################

    # Connection upgrade variable for websocket
    map $http_upgrade $connection_upgrade {  
        default upgrade;
        ''      close;
    }

    server {
        server_name localhost 127.0.0.1;
        listen 8080 ssl http2;

        location / {
            proxy_pass   http://127.0.0.1:8051;
        }

        location /api {
            # Websocket headers
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $http_host;
            proxy_pass   http://127.0.0.1:8051;
        }
    }
}
  1. Setup local certificates with mkcert (otherwise grpc/websockets won't work) and assign correct permissions:
sudo mkcert -install && sudo mkcert localhost 127.0.0.1 && sudo chown $USER:$USER *.pem
  1. Launch nginx:
nginx -p $PWD -e stderr -c nginx.conf

sshx setup

The default sshx binaries won't work, because tonic is compiled with tls-webpki-roots feature, which means that sshx will not lookup OS ca-certificates bundle, making our self-signed certificates useless and throwing the following error:

❯ ./sshx --server https://localhost:8080
2023-11-06T12:15:26.058671Z ERROR sshx: transport error

Caused by:
    0: error trying to connect: invalid peer certificate: UnknownIssuer
    1: invalid peer certificate: UnknownIssuer

In order to fix that we need to recompile sshx binary with tls-roots feature (which will lookup OS ca-certificates):

  1. Enter sshx cloned git repository:
cd /tmp/reproduce-issue/sshx
  1. Save the following content as tonic_fix.patch:
diff --git a/Cargo.toml b/Cargo.toml
index 4b0c58e..c028b8d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -19,7 +19,7 @@ rand = "0.8.5"
 serde = { version = "1.0.188", features = ["derive", "rc"] }
 tokio = { version = "1.32.0", features = ["full"] }
 tokio-stream = { version = "0.1.14", features = ["sync"] }
-tonic = { version = "0.10.0", features = ["tls", "tls-webpki-roots"] }
+tonic = { version = "0.10.0", features = ["tls", "tls-roots"] }
 tracing = "0.1.37"
 tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }

and apply it like so:

git apply tonic_fix.patch
  1. Compile modified sshx:
cargo build --release --bin sshx
  1. Launch client:
mv target/release/sshx . && ./sshx --server https://localhost:8080

Now, here's the issue: there's no connection between server and shell and I'm unable to spawn terminal:

image

And once in a while sshx produces the following log:

2023-11-06T14:25:05.152762Z ERROR sshx::controller: disconnected, retrying in 1s... err=status: Internal, message: "h2 protocol error: http2 error: stream error received: unspecific protocol error detected", details: [], metadata: MetadataMap { headers: {} }

Here's the log with RUST_LOG=trace: trace_sshx.log

Any help would be much appreciated!

Hugo1380 commented

Adding this to the server section seems to fix it:

location /sshx.SshxService/Channel {
   grpc_pass      grpc://127.0.0.1:8051;
}
shumvgolove commented

Yep, this directive fixes the issue. Thanks a lot!

Maybe we should add the working nginx.conf example to wiki?

  • 👍2
natarajan0007 commented

release]$ ./sshx --server https://localhost:8003
2023-11-07T19:15:11.596704Z ERROR sshx: transport error
image

Caused by:
0: error trying to connect: invalid peer certificate: UnknownIssuer
1: invalid peer certificate: UnknownIssuer
[muthUn@brlb release]$

As per your suggestion i was changed the below line of code in cargo.toml file still i am getting certificate error message.

diff --git a/Cargo.toml b/Cargo.toml index 4b0c58e..c028b8d 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -19,7 +19,7 @@ rand = "0.8.5" serde = { version = "1.0.188", features = ["derive", "rc"] } tokio = { version = "1.32.0", features = ["full"] } tokio-stream = { version = "0.1.14", features = ["sync"] } -tonic = { version = "0.10.0", features = ["tls", "tls-webpki-roots"] } +tonic = { version = "0.10.0", features = ["tls", "tls-roots"] } tracing = "0.1.37" tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }

@shumvgolove

ekzhang commented

Yes, you'll have to grpc_pass all requests to suffixes of /sshx.SshxService.

Actually on the server it internally just checks if the content-type is application/grpc to decide whether to handle it as an HTTP request or a gRPC request. But since all gRPC requests go to SshxService, matching by path would work fine.

sshx/crates/sshx-server/src/listen.rs

Lines 51 to 60 in e01b75a

let svc = Steer::new(
[http_service, grpc_service],
|req: &Request<Body>, _services: &[_]| {
let headers = req.headers();
match headers.get(CONTENT_TYPE) {
Some(content) if content == "application/grpc" => 1,
_ => 0,
}
},
);

To clarify, you don't necessarily need to detect gRPC when you're putting a reverse proxy in front of sshx. You just need to forward HTTP/2 connections to it as the backend protocol. But Nginx doesn't let you do h2c (HTTP/2 cleartext) backend for its reverse proxy; it only has grpc_pass as a special case.

Sorry about this, Nginx is just eternally difficult to configure 😅

  • 👍2
natarajan0007 commented

this is what i have in my nginx.conf . I am still facing certificate issue. would you mind sharing your nginx setup @shumvgolove

$ ./sshx --server https://localhost:8002
2023-11-09T09:39:28.982734Z ERROR sshx: transport error

Caused by:
0: error trying to connect: invalid peer certificate: UnknownIssuer
1: invalid peer certificate: UnknownIssuer

`events {}

pid nginx.pid;

http {
access_log /dev/stdout;

ssl_certificate ./localhost.pem;
ssl_certificate_key ./localhost-key.pem;
ssl_protocols	TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

client_body_temp_path .;
proxy_temp_path .;
fastcgi_temp_path .;
uwsgi_temp_path .;
scgi_temp_path .;

client_body_timeout 5;

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    server_name localhost 127.0.0.1;
    listen 8003 ssl http2;

    location /sshx.SshxService {
        grpc_pass      grpc://127.0.0.1:8051;
    }

    location /api {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $http_host;
        proxy_pass   http://127.0.0.1:8051;
    }
}

}
`

natarajan0007 commented

I tried without nginx . Its working fine. But i will try to see if i can able to make with nginx.

image

juiiyang commented

Hey 👋 and thanks for the great project!

After several attempts to figure out how to make it work with nginx reverse-proxy, I've created half-working solution. I am providing below steps to reproduce the issue and ultimately figure out what is the problem.

sshx-server setup

  1. Create and enter temp folder:
mkdir /tmp/reproduce-issue && cd /tmp/reproduce-issue
  1. Clone and enter the sshx repository:
git clone https://github.com/ekzhang/sshx.git && cd sshx
  1. Build frontend:
npm ci && npm run build
  1. Build backend:
cargo build --release --bin sshx-server
  1. Launch sshx (which will also serve frontend).:
mv target/release/sshx-server . && ./sshx-server --secret dev-secret --listen :: --override-origin https://localhost:8080

nginx setup

  1. Create and enter nginx folder:
mkdir /tmp/reproduce-issue/nginx && cd /tmp/reproduce-issue/nginx
  1. Create nginx.conf with the following content:
# Run nginx using:
#     nginx -p $PWD -e stderr -c nginx.conf

daemon off;  # run in foreground

events {}

pid nginx.pid;

http {
    access_log /dev/stdout;

    # Locally generated SSL certificates from mkcert
    ssl_certificate ./localhost+1.pem;
    ssl_certificate_key ./localhost+1-key.pem;
    ssl_protocols	TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

    # Directories nginx needs configured to start up.
    client_body_temp_path .;
    proxy_temp_path .;
    fastcgi_temp_path .;
    uwsgi_temp_path .;
    scgi_temp_path .;
    
    ###################
    client_body_timeout 5;  # <------ makes sshx client spitting error log every 5 seconds
    ###################

    # Connection upgrade variable for websocket
    map $http_upgrade $connection_upgrade {  
        default upgrade;
        ''      close;
    }

    server {
        server_name localhost 127.0.0.1;
        listen 8080 ssl http2;

        location / {
            proxy_pass   http://127.0.0.1:8051;
        }

        location /api {
            # Websocket headers
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $http_host;
            proxy_pass   http://127.0.0.1:8051;
        }
    }
}
  1. Setup local certificates with mkcert (otherwise grpc/websockets won't work) and assign correct permissions:
sudo mkcert -install && sudo mkcert localhost 127.0.0.1 && sudo chown $USER:$USER *.pem
  1. Launch nginx:
nginx -p $PWD -e stderr -c nginx.conf

sshx setup

The default sshx binaries won't work, because tonic is compiled with tls-webpki-roots feature, which means that sshx will not lookup OS ca-certificates bundle, making our self-signed certificates useless and throwing the following error:

❯ ./sshx --server https://localhost:8080
2023-11-06T12:15:26.058671Z ERROR sshx: transport error

Caused by:
    0: error trying to connect: invalid peer certificate: UnknownIssuer
    1: invalid peer certificate: UnknownIssuer

In order to fix that we need to recompile sshx binary with tls-roots feature (which will lookup OS ca-certificates):

  1. Enter sshx cloned git repository:
cd /tmp/reproduce-issue/sshx
  1. Save the following content as tonic_fix.patch:
diff --git a/Cargo.toml b/Cargo.toml
index 4b0c58e..c028b8d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -19,7 +19,7 @@ rand = "0.8.5"
 serde = { version = "1.0.188", features = ["derive", "rc"] }
 tokio = { version = "1.32.0", features = ["full"] }
 tokio-stream = { version = "0.1.14", features = ["sync"] }
-tonic = { version = "0.10.0", features = ["tls", "tls-webpki-roots"] }
+tonic = { version = "0.10.0", features = ["tls", "tls-roots"] }
 tracing = "0.1.37"
 tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }

and apply it like so:

git apply tonic_fix.patch
  1. Compile modified sshx:
cargo build --release --bin sshx
  1. Launch client:
mv target/release/sshx . && ./sshx --server https://localhost:8080

Now, here's the issue: there's no connection between server and shell and I'm unable to spawn terminal:

image

And once in a while sshx produces the following log:

2023-11-06T14:25:05.152762Z ERROR sshx::controller: disconnected, retrying in 1s... err=status: Internal, message: "h2 protocol error: http2 error: stream error received: unspecific protocol error detected", details: [], metadata: MetadataMap { headers: {} }

Here's the log with RUST_LOG=trace: trace_sshx.log

Any help would be much appreciated!

Thanks for your guide, my problem is solved.

By the way, self signed cert need to add into keychain in Mac OS