SSH client warning "server gave bad signature"
vholer opened this issue · 8 comments
When having upstream OpenSSH server (6.8+), which is using different SSH host keys than sshpiper, the upstream server notifies the client as part of SSH protocol about its different server host keys (extension hostkeys-00@openssh.com
). By default, OpenSSH client tries to deal with them which results in a warning about wrong signature.
$ ssh app1@ssh.example.com
client_global_hostkeys_private_confirm: server gave bad signature for ECDSA key 0: incorrect signature
app1:~$
Problem can be silenced on client by ignoring the server hostkeys notifications via ssh client option:
$ ssh -o 'UpdateHostkeys no' app1@ssh.example.com
app1:~$
Observations:
-
current OpenSSH servers always notify clients, this extension can't be disabled in configuration https://github.com/openssh/openssh-portable/blob/4a5590a5ee47b7dfd49773e9fdba48ad3089fe64/sshd.c#L2304
-
Dropbear (different SSH server) doesn't support this protocol extensions for now thus clients don't trigger this warning
I'm not an expert on SSH nor sshpiper architecture, so I'm not sure if there would be an easy solution. If sshpiper is aware of the SSH protocol when communicating with the upstream server, it would be nice if it could filter on the SSH packet type 80 with the hostkeys-00@openssh.com
:
we can filter it out,
I am checking why i did not see it in 8.8
Maybe you are using/testing that with custom known hosts file? Than the update of host keys is disabled on client.
$ man ssh_config
...
UpdateHostKeys is enabled by default if the user has not overridden the de‐
fault UserKnownHostsFile setting and has not enabled VerifyHostKeyDNS, oth‐
erwise UpdateHostKeys will be set to no.
Also, you might already have fingerprints of both sshpiper and upstream server in known hosts already, but under different names. With this configuration, ssh client works without writing any warning, although the configuration is not completely correct. I.e.,
$ cat ~/.ssh/known_hosts
ssh.example.com ecdsa-sha2-nistp256 XXXXXXXXXXXXXXXXXX
app1.example.com ssh-ed25519 YYYYYYYYYYYYYYY
$ ssh -vvv app1@ssh.example.com
...
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: client_input_hostkeys: received RSA key SHA256:9KW7OfsCS01nnoW4eG11kfSI74fme8nCFvjcbgrmGqc
debug3: client_input_hostkeys: received ECDSA key SHA256:dGhQjEiK4n6/B/luYUt9rEl1iplrwXNJ26lR0oIMUHU
debug3: client_input_hostkeys: received ED25519 key SHA256:V6/1jZ6sO2lhrs0A0Upm2/BJ+LBhV/MRLWRxKfKGtKs
debug1: client_input_hostkeys: searching /root/.ssh/known_hosts for ssh.example.com / (none)
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: hostkeys_find: deprecated ecdsa-sha2-nistp256 key at /root/.ssh/known_hosts:1
!!! debug3: hostkeys_find: found ssh-ed25519 key under different name/addr at /root/.ssh/known_hosts:2
debug1: client_input_hostkeys: searching /root/.ssh/known_hosts2 for ssh.example.com / (none)
debug1: client_input_hostkeys: hostkeys file /root/.ssh/known_hosts2 does not exist
debug3: client_input_hostkeys: 3 server keys: 3 new, 18446744073709551613 retained, 3 incomplete match. 1 to remove
!!! debug1: client_input_hostkeys: host key found matching a different name/address, skipping UserKnownHostsFile update
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
...
now I can repro by ssh -o 'UpdateHostkeys yes'
this issue is very interesting.
sshpiper can filter all ssh pkg and here is the usage now https://github.com/tg123/sshpiper#screening-recording
however, it is too expensive to filter all pkgs, I am thinking how to filter it out with some better way
filtering hostkeys-00@openssh.com
and hostkeys-prove-00@openssh.com
works but seems cost too much.
Very nice! How much is "too much" (%cpu overhead, -%network throughput)? I guess for interactive SSH use, the affect won't be much visible, but the concern are sessions with large network traffic (sfp, rsync, ...).
Could this filtering be only for a limited time? E.g.,
- stop any further filtering after the hostkeys/prove are processed (I guess no further keys notification should arrive later again),
- or if server doesn't initially notify, hook on some other relevant SSH packet at the end of new session opening,
- or silly disable after timeout.
you check my private branch here
master...filter
I have tested the changes (with 3 VMs on Linode, all dedicated 2 VCPU - ssh client, sshpiper, ssh upstream server) and I don't see significant degradation in the filter branch 👍 . So, it looks good to me. I have validated that stable sshpiper is complaning about the SSH server signature problem and the patched one doesn't (thanks!), so I'm sure I was using the right one.
scp throughput
- direct connection - 152.7 MiB/s
- sshpiper (v1.0.83) - 122.6 MiB/s, 152.54 % CPU (77.84 %usr, 74.70 %system, 2.66 %wait)
- sshpiper (filter) - 122.1 MiB/s, 153.14 % CPU (77.92 %usr, 75.22 %system, 3.31 %wait)
For both sshpipers, the throughput degradation compared to direct connection is 20%. On CPU also both sshpipers consume same.
sshping
Used https://github.com/spook/sshping, executed as sshping -P -c 50000 -s 1000 root@sshpiper-server
.
- direct connection
ssh-Login-Time: 1627985707 ns
Minimum-Latency: 145022 ns
Median-Latency: 174882 ns
Average-Latency: 178701 ns
Average-Deviation: 44723 ns
Maximum-Latency: 6880441 ns
Echo-Count: 50000 B
rtt min/avg/max/mdev = 0.145/0.178/6.880/0.044 ms
Upload-Size: 1000000000 B
Upload-Rate: 291950697 B/s
Download-Size: 1000000000 B
Download-Rate: 9536275 B/s
- sshpiper (v1.0.83)
ssh-Login-Time: 1666912934 ns
Minimum-Latency: 304125 ns
Median-Latency: 372917 ns
Average-Latency: 376017 ns
Average-Deviation: 48467 ns
Maximum-Latency: 7124137 ns
Echo-Count: 50000 B
rtt min/avg/max/mdev = 0.304/0.376/7.124/0.048 ms
Upload-Size: 1000000000 B
Upload-Rate: 145001499 B/s
Download-Size: 1000000000 B
Download-Rate: 9288293 B/s
- sshpiper (filter)
ssh-Login-Time: 1668404632 ns
Minimum-Latency: 318176 ns
Median-Latency: 374956 ns
Average-Latency: 377973 ns
Average-Deviation: 42268 ns
Maximum-Latency: 4869657 ns
Echo-Count: 50000 B
rtt min/avg/max/mdev = 0.318/0.377/4.869/0.042 ms
Upload-Size: 1000000000 B
Upload-Rate: 140450266 B/s
Download-Size: 1000000000 B
Download-Rate: 9283925 B/s
So to summarize - latencies to get back text sent via echo:
- direct: rtt min/avg/max/mdev = 0.145/0.178/6.880/0.044 ms
- sshpiper (v1.0.83) rtt min/avg/max/mdev = 0.304/0.376/7.124/0.048 ms
- sshpiper (filter) rtt min/avg/max/mdev = 0.318/0.377/4.869/0.042 ms
Average latencies for both sshpipers are same, in general sshpiper increases latency by 111% (NOTE: average ICMP ping latency between each server is 0.452 ms, so I don't quite understand how they measure it so that sshping latencies are even lower, hmmm).
With sshpiper, upload throughput is degraded by 51% and download (this test must be broken) only by 3%.
awesome benchmark
seems it is not a big deal to add a filter to every single pkg
i will make it an option.