Segfaults when running copy_remote_file on proxy_client
maxballenger opened this issue · 2 comments
For general questions please use the mail group.
Describe the bug
Hi @pkittenis , following up on #292 as I think I encountered a similar issue. I am trying to connect to a single host behind a proxy and do put/get files and run commands on that host. I am also trying to do put/get files and run commands on the proxy itself. I had been doing this with two separate SSHClient() objects as shown in #292. I refactored my code to only use a single SSHClient object and to talk to the proxy using its _proxy_client attribute. However, this is not working for me. I am getting frequent segfaults. I've attached a test to reproduce.
To Reproduce
Steps to reproduce the behavior:
- Fill in the constants in the attached test with appropriate IP addresses, username, and paths to keys
- Run the test
- Typically "test_2_many_getfile_host" will proceed just fine, but "test_2_many_getfile_proxy" will not make it through 20 iterations, regardless of whether it runs before or after the host test.
- I get many different error messages, some of which are shown below.
- python3: /libssh2-1.9.0.tar.gz/libssh2-libssh2-1.9.0/src/transport.c:349: _libssh2_transport_read: Assertion `remainbuf >= 0' failed.
- Silent python exit but dmesg indicates something like the following (not always in libcrypto) "python3[12794]: segfault at 1cdb000 ip 00007f7006cae3cd sp 00007f70025445c0 error 4 in libcrypto-ab1baf6c.so.1.1[7f7006ad2000+2b0000]"
Here is a GDB backtrace I got from running the test under GDB when it exited with the first message above about remainbuf:
Thread 1 "python3" received signal SIGABRT, Aborted.
0x00007f7e1b3b5fb7 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007f7e1b3b5fb7 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f7e1b3b7921 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f7e1b3a748a in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f7e1b3a7502 in __assert_fail ()
from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f7e1883e677 in _libssh2_transport_read ()
from /usr/lib/python3.6/site-packages/ssh2/../ssh2_python.libs/libssh2-55f304ca.so.1.0.1
#5 0x00007f7e18817dd3 in _libssh2_channel_write ()
from /usr/lib/python3.6/site-packages/ssh2/../ssh2_python.libs/libssh2-55f304ca.so.1.0.1
#6 0x00007f7e18838b6e in sftp_open ()
from /usr/lib/python3.6/site-packages/ssh2/../ssh2_python.libs/libssh2-55f304ca.so.1.0.1
#7 0x00007f7e18839157 in libssh2_sftp_open_ex ()
from /usr/lib/python3.6/site-packages/ssh2/../ssh2_python.libs/libssh2-55f304ca.so.1.0.1
#8 0x00007f7e16fcac1a in __pyx_pf_4ssh2_4sftp_4SFTP_8open (
__pyx_v_mode=256, __pyx_v_flags=1, __pyx_v_filename=<optimized out>,
__pyx_v_self=0x7f7e15279be0) at ssh2/sftp.c:2381
#9 __pyx_pw_4ssh2_4sftp_4SFTP_9open (__pyx_v_self=0x7f7e15279be0,
__pyx_args=<optimized out>, __pyx_kwds=<optimized out>)
Expected behavior
Able to disconnect/reconnect from proxy and still get files
Actual behaviour
Segfaults when I try to disconnect/reconnect from proxy and get files
Screenshots
If applicable, add screenshots to help explain your problem.
Additional information
ssh2_python==0.22.0
parallel-ssh==2.4.5
Code to Reproduce
import unittest
import time
from pssh.clients import SSHClient
PROXY_IP = '10.1.15.1'
HOST_IP = '172.16.0.101'
PROXY_USER = 'user'
HOST_USER = 'user'
PROXY_KEY = "/path/to/proxy/key"
HOST_KEY = "/path/to/host/key"
TEST_FILE = "test_file.txt"
RECONNECT_DELAY = 1
NUMBER_RECONNECTS = 20
class TestB1Sensor(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.ssh = HostAndProxy(PROXY_IP, HOST_IP, PROXY_USER, HOST_USER, PROXY_KEY, HOST_KEY)
def get_file(self, host):
host.get_file(TEST_FILE, TEST_FILE)
with open(TEST_FILE, 'r') as f:
print(f.read())
def test_1_many_getfile_proxy(self):
print("Testing getfile on proxy")
for i in range(NUMBER_RECONNECTS):
print(f"Running cycle {i+1}/{NUMBER_RECONNECTS}")
print("Closing ssh connection")
self.ssh.disconnect()
time.sleep(RECONNECT_DELAY)
self.ssh.connect()
print("Connection re-established")
self.get_file(self.ssh.proxy)
def test_2_many_getfile_host(self):
print("Testing getfile on host")
for i in range(NUMBER_RECONNECTS):
print(f"Running cycle {i}/{NUMBER_RECONNECTS}")
print("Closing ssh connection")
self.ssh.disconnect()
time.sleep(RECONNECT_DELAY)
self.ssh.connect()
print("Connection re-established")
self.get_file(self.ssh.host)
class HostAndProxy:
def __init__(self, ip_proxy, ip_host, user_proxy, user_host, key_proxy, key_host):
self.ip_proxy = ip_proxy
self.ip_host = ip_host
self.user_proxy = user_proxy
self.user_host = user_host
self.key_proxy = key_proxy
self.key_host = key_host
self.connect()
def connect(self):
self.host = Host(self.ip_proxy, self.ip_host,
self.user_proxy, self.user_host,
self.key_proxy, self.key_host)
self.proxy = Proxy(self.host.get_proxy_client())
def disconnect(self):
self.host.disconnect()
class SSHTarget:
def get_file(self, remotepath, localpath, recurse: bool = False):
self.ssh.copy_remote_file(remotepath, localpath, recurse=recurse)
class Host(SSHTarget):
def __init__(self, ip_proxy, ip_host, user_proxy, user_host, key_proxy, key_host):
# Configure IP
self.ip_proxy = ip_proxy
self.ip_host = ip_host
self.user_proxy = user_proxy
self.user_host = user_host
self.key_proxy = key_proxy
self.key_host = key_host
self.connect()
def connect(self):
kwargs = {
'host': str(self.ip_host),
'user': self.user_host,
'pkey': self.key_host,
'proxy_host': str(self.ip_proxy),
'proxy_user': self.user_proxy,
'proxy_pkey': self.key_proxy
}
self.ssh = SSHClient(**kwargs)
def disconnect(self):
self.ssh.disconnect()
def get_proxy_client(self) -> SSHClient:
return self.ssh._proxy_client
class Proxy(SSHTarget):
def __init__(self, proxy_client):
self.ssh = proxy_client
if __name__ == '__main__':
unittest.main()
OK, after poking through the library code a little more, using the _proxy_client attribute the way I am seems like a bad idea. However, if I don't do it this way, then I am back in the land of creating two SSHClient() objects, one for the end host through and one for the proxy itself, if I want to do SSH ops on both the proxy and the endpoint host. This also creates segfaults.
Closing this as I've narrowed it down and this issue report has a lot of irrelevant information in it