PoonLab/covizu

epicov: File transfer failing in batch.py script

Closed this issue · 4 comments

GopiGugan commented 
[covizu@Paphlagon ~]$ tail batch.log 
R[write to console]:  library ‘/usr/share/R/library’ contains no packages

ssh: connect to host filogeneti.ca port 22: Connection timed out
scp: Connection closed
Traceback (most recent call last):
  File "/home/covizu/covizu/batch.py", line 214, in <module>
    subprocess.check_call(['scp', nwk_file, '{}/timetree.nwk'.format(server_epicov)])
  File "/usr/lib64/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['scp', 'data/timetree.2023-10-21T06:22:09.nwk', 'filogeneti.ca:/var/www/html/epicov/data/timetree.nwk']' returned non-zero exit status 255.
GopiGugan commented 
[covizu@Paphlagon covizu]$ python3
Python 3.11.3 (main, Apr  5 2023, 00:00:00) [GCC 12.2.1 20221121 (Red Hat 12.2.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_call(['scp', 'data/timetree.2023-10-21T06:22:09.nwk', 'filogeneti.ca:/var/www/html/epicov/data/timetree_test.nwk'])
timetree.2023-10-21T06:22:09.nwk                                                                                                                                                           100%   69KB  29.5MB/s   00:00    
0
GopiGugan commented

There is a rule in place on the webserver to prevent SSH brute force attacks.

As a result, we are running into the SSH timeout when running multiple scp commands sequentially:

covizu/batch.py

Lines 208 to 215 in 4ab1ba4

subprocess.check_call(['scp', nwk_file, '{}/timetree.nwk'.format(server_root)])
subprocess.check_call(['scp', clust_file, '{}/clusters.json'.format(server_root)])
subprocess.check_call(['scp', dbstat_file, '{}/dbstats.json'.format(server_root)])
# upload files to EpiCoV server
server_epicov = 'filogeneti.ca:/var/www/html/epicov/data'
subprocess.check_call(['scp', nwk_file, '{}/timetree.nwk'.format(server_epicov)])
subprocess.check_call(['scp', dbstat_file, '{}/dbstats.json'.format(server_epicov)])

ArtPoon commented

Let's add time.sleep(5) commands between calls to scp in the local copy of batch.py temporarily until we switch from the VM webserver back to ccr5.

ArtPoon commented

Moved back to original webserver, this isn't an issue anymore