tg123/sshpiper

SSH client warning "server gave bad signature"

vholer opened this issue · 8 comments

vholer commented

When having upstream OpenSSH server (6.8+), which is using different SSH host keys than sshpiper, the upstream server notifies the client as part of SSH protocol about its different server host keys (extension hostkeys-00@openssh.com). By default, OpenSSH client tries to deal with them which results in a warning about wrong signature.

$ ssh app1@ssh.example.com
client_global_hostkeys_private_confirm: server gave bad signature for ECDSA key 0: incorrect signature
app1:~$

Problem can be silenced on client by ignoring the server hostkeys notifications via ssh client option:

$ ssh -o 'UpdateHostkeys no' app1@ssh.example.com
app1:~$

Observations:

I'm not an expert on SSH nor sshpiper architecture, so I'm not sure if there would be an easy solution. If sshpiper is aware of the SSH protocol when communicating with the upstream server, it would be nice if it could filter on the SSH packet type 80 with the hostkeys-00@openssh.com:

https://github.com/openssh/openssh-portable/blob/4a5590a5ee47b7dfd49773e9fdba48ad3089fe64/PROTOCOL#L291-L304

tg123 commented

we can filter it out,
I am checking why i did not see it in 8.8

vholer commented

Maybe you are using/testing that with custom known hosts file? Than the update of host keys is disabled on client.

$ man ssh_config
...
UpdateHostKeys is enabled by default if the user has not overridden the de‐
fault UserKnownHostsFile setting and has not enabled VerifyHostKeyDNS, oth‐
erwise UpdateHostKeys will be set to no.

Also, you might already have fingerprints of both sshpiper and upstream server in known hosts already, but under different names. With this configuration, ssh client works without writing any warning, although the configuration is not completely correct. I.e.,

    $ cat ~/.ssh/known_hosts
    ssh.example.com ecdsa-sha2-nistp256 XXXXXXXXXXXXXXXXXX
    app1.example.com ssh-ed25519 YYYYYYYYYYYYYYY

    $ ssh -vvv app1@ssh.example.com
    ...
    debug3: receive packet: type 80
    debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
    debug3: client_input_hostkeys: received RSA key SHA256:9KW7OfsCS01nnoW4eG11kfSI74fme8nCFvjcbgrmGqc
    debug3: client_input_hostkeys: received ECDSA key SHA256:dGhQjEiK4n6/B/luYUt9rEl1iplrwXNJ26lR0oIMUHU
    debug3: client_input_hostkeys: received ED25519 key SHA256:V6/1jZ6sO2lhrs0A0Upm2/BJ+LBhV/MRLWRxKfKGtKs
    debug1: client_input_hostkeys: searching /root/.ssh/known_hosts for ssh.example.com / (none)
    debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
    debug3: hostkeys_find: deprecated ecdsa-sha2-nistp256 key at /root/.ssh/known_hosts:1
!!! debug3: hostkeys_find: found ssh-ed25519 key under different name/addr at /root/.ssh/known_hosts:2
    debug1: client_input_hostkeys: searching /root/.ssh/known_hosts2 for ssh.example.com / (none)
    debug1: client_input_hostkeys: hostkeys file /root/.ssh/known_hosts2 does not exist
    debug3: client_input_hostkeys: 3 server keys: 3 new, 18446744073709551613 retained, 3 incomplete match. 1 to remove
!!! debug1: client_input_hostkeys: host key found matching a different name/address, skipping UserKnownHostsFile update
    debug3: receive packet: type 91
    debug2: channel_input_open_confirmation: channel 0: callback start
...
tg123 commented

now I can repro by ssh -o 'UpdateHostkeys yes'
this issue is very interesting.

sshpiper can filter all ssh pkg and here is the usage now https://github.com/tg123/sshpiper#screening-recording

however, it is too expensive to filter all pkgs, I am thinking how to filter it out with some better way

tg123 commented

filtering hostkeys-00@openssh.com and hostkeys-prove-00@openssh.com works but seems cost too much.

vholer commented

Very nice! How much is "too much" (%cpu overhead, -%network throughput)? I guess for interactive SSH use, the affect won't be much visible, but the concern are sessions with large network traffic (sfp, rsync, ...).

Could this filtering be only for a limited time? E.g.,

  • stop any further filtering after the hostkeys/prove are processed (I guess no further keys notification should arrive later again),
  • or if server doesn't initially notify, hook on some other relevant SSH packet at the end of new session opening,
  • or silly disable after timeout.
tg123 commented

you check my private branch here
master...filter

vholer commented

I have tested the changes (with 3 VMs on Linode, all dedicated 2 VCPU - ssh client, sshpiper, ssh upstream server) and I don't see significant degradation in the filter branch 👍 . So, it looks good to me. I have validated that stable sshpiper is complaning about the SSH server signature problem and the patched one doesn't (thanks!), so I'm sure I was using the right one.

scp throughput

  • direct connection - 152.7 MiB/s
  • sshpiper (v1.0.83) - 122.6 MiB/s, 152.54 % CPU (77.84 %usr, 74.70 %system, 2.66 %wait)
  • sshpiper (filter) - 122.1 MiB/s, 153.14 % CPU (77.92 %usr, 75.22 %system, 3.31 %wait)

For both sshpipers, the throughput degradation compared to direct connection is 20%. On CPU also both sshpipers consume same.

sshping

Used https://github.com/spook/sshping, executed as sshping -P -c 50000 -s 1000 root@sshpiper-server.

  • direct connection
ssh-Login-Time:         1627985707 ns
Minimum-Latency:            145022 ns
Median-Latency:             174882 ns
Average-Latency:            178701 ns
Average-Deviation:           44723 ns
Maximum-Latency:           6880441 ns
Echo-Count:                  50000 B
rtt min/avg/max/mdev = 0.145/0.178/6.880/0.044 ms
Upload-Size:            1000000000 B
Upload-Rate:             291950697 B/s
Download-Size:          1000000000 B
Download-Rate:             9536275 B/s
  • sshpiper (v1.0.83)
ssh-Login-Time:         1666912934 ns
Minimum-Latency:            304125 ns
Median-Latency:             372917 ns
Average-Latency:            376017 ns
Average-Deviation:           48467 ns
Maximum-Latency:           7124137 ns
Echo-Count:                  50000 B
rtt min/avg/max/mdev = 0.304/0.376/7.124/0.048 ms
Upload-Size:            1000000000 B
Upload-Rate:             145001499 B/s
Download-Size:          1000000000 B
Download-Rate:             9288293 B/s
  • sshpiper (filter)
ssh-Login-Time:         1668404632 ns
Minimum-Latency:            318176 ns
Median-Latency:             374956 ns
Average-Latency:            377973 ns
Average-Deviation:           42268 ns
Maximum-Latency:           4869657 ns
Echo-Count:                  50000 B
rtt min/avg/max/mdev = 0.318/0.377/4.869/0.042 ms
Upload-Size:            1000000000 B
Upload-Rate:             140450266 B/s
Download-Size:          1000000000 B
Download-Rate:             9283925 B/s

So to summarize - latencies to get back text sent via echo:

- direct:              rtt min/avg/max/mdev = 0.145/0.178/6.880/0.044 ms
- sshpiper (v1.0.83)   rtt min/avg/max/mdev = 0.304/0.376/7.124/0.048 ms
- sshpiper (filter)    rtt min/avg/max/mdev = 0.318/0.377/4.869/0.042 ms

Average latencies for both sshpipers are same, in general sshpiper increases latency by 111% (NOTE: average ICMP ping latency between each server is 0.452 ms, so I don't quite understand how they measure it so that sshping latencies are even lower, hmmm).

With sshpiper, upload throughput is degraded by 51% and download (this test must be broken) only by 3%.

tg123 commented

awesome benchmark
seems it is not a big deal to add a filter to every single pkg
i will make it an option.

  • ❤️1