Encountered ConnectionRefusedError, so killing all jobs
nick-youngblut opened this issue · 4 comments
nick-youngblut commented
The 2 example scripts were working fine after adding the following:
os.environ['SGE_ROOT'] = '/var/lib/gridengine'
os.environ['SGE_CELL'] = 'default'
os.environ['DRMAA_LIBRARY_PATH'] = '/usr/lib/gridengine-drmaa/lib/libdrmaa.so.1.0'
os.environ['ERROR_MAIL_SENDER'] = 'False'
os.environ['CREATE_PLOTS'] = 'False'
That is, until this last weekend, when a power outage damaged a switch in the cluster. The cluster was rebooted, and since then, I get the following error:
2020-04-21 13:06:54,380 - gridmap.job - INFO - Encountered ConnectionRefusedError, so killing all jobs.
Traceback (most recent call last):
File "../deepmased/DeepMAsED-SM/tmp/gridmap_manual.py", line 94, in <module>
main()
File "../deepmased/DeepMAsED-SM/tmp/gridmap_manual.py", line 86, in main
temp_dir='/ebio/abt3_scratch/gridmap_tmp/')
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/site-packages/gridmap/job.py", line 897, in process_jobs
monitor.check(sid, jobs)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/site-packages/gridmap/job.py", line 452, in check
self.check_if_alive()
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/site-packages/gridmap/job.py", line 500, in check_if_alive
send_error_mail(job)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/site-packages/gridmap/job.py", line 675, in send_error_mail
_send_mail(subject, body_text, attachments)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/site-packages/gridmap/job.py", line 565, in _send_mail
s = smtplib.SMTP(SMTP_SERVER)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/smtplib.py", line 336, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/smtplib.py", line 307, in _get_socket
self.source_address)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/socket.py", line 724, in create_connection
raise err
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
I tried printing the job.ret attribute in the check_if_alive() function, and I got:
=====================================
======== Submit and Wait ========
=====================================
sending function jobs to cluster
2020-04-21 13:06:38,967 - gridmap.job - INFO - Setting up JobMonitor on tcp://172.18.3.170:45659
*@#%$*@#___GRIDMAP___NOT___DONE___@#%**#*$&*%
*@#%$*@#___GRIDMAP___NOT___DONE___@#%**#*$&*%
*@#%$*@#___GRIDMAP___NOT___DONE___@#%**#*$&*%
*@#%$*@#___GRIDMAP___NOT___DONE___@#%**#*$&*%
2020-04-21 13:06:49,752 - gridmap.job - INFO - Received exception from 9455057
2020-04-21 13:06:51,808 - gridmap.job - INFO - Received exception from 9455056
2020-04-21 13:06:52,924 - gridmap.job - INFO - Received exception from 9455054
2020-04-21 13:06:53,669 - gridmap.job - INFO - Received exception from 9455055
'Job' object has no attribute '_f'
2020-04-21 13:06:54,380 - gridmap.job - INFO - Encountered ConnectionRefusedError, so killing all jobs.
I've talked to the cluster admin about this, and he can't think of anything that could be causing the issue. Any ideas?
mulhod commented
I'm not sure right now, but I see the following in conf.py:
# Global settings ####
# email settings
SEND_ERROR_MAIL = 'TRUE' == os.getenv('SEND_ERROR_MAIL', 'True').upper()
SMTP_SERVER = os.getenv('SMTP_SERVER', '.'.join(gethostname().split('.')[-3:]))
ERROR_MAIL_SENDER = os.getenv('ERROR_MAIL_SENDER', 'error@gridmap.py')
ERROR_MAIL_RECIPIENT = os.getenv('ERROR_MAIL_RECIPIENT',
'{}@{}'.format(os.getenv('USER'),
'.'.join(gethostname().split('.')[-2:])))In your code sample, you're setting ERROR_MAIL_SENDER to False, but I think that that should be for SEND_ERROR_MAIL, actually. You don't want to send error mail, right? I'm not sure this will resolve your issue, but it's worth a shot at least for debugging purposes.
nick-youngblut commented
Thanks @mulhod for your suggestion! That didn't work. Maybe it would help to post the entire output:
full log
2020-04-21 17:29:11,984 - Setting up JobMonitor on tcp://172.18.3.170:41971
Your job gridmap_job0 has been submitted with id 9459727
Your job gridmap_job1 has been submitted with id 9459728
Your job gridmap_job2 has been submitted with id 9459729
Your job gridmap_job3 has been submitted with id 9459730
2020-04-21 17:29:12,009 - Starting local hearbeat
2020-04-21 17:29:12,018 - Starting ZMQ event loop
2020-04-21 17:29:12,018 - 0 out of 4 jobs completed
2020-04-21 17:29:12,018 - Waiting for message
2020-04-21 17:29:12,020 - Connecting to JobMonitor (tcp://172.18.3.170:41971)
2020-04-21 17:29:12,021 - Sending message: {'job_id': -1, 'host_name': 'rick', 'ip_address': '172.18.3.170', 'command': 'heart_beat', 'data': {}}
2020-04-21 17:29:12,021 - Received message: {'job_id': -1, 'host_name': 'rick', 'ip_address': '172.18.3.170', 'command': 'heart_beat', 'data': {}}
2020-04-21 17:29:12,022 - Checking if jobs are alive
2020-04-21 17:29:12,022 - Sending reply:
2020-04-21 17:29:12,022 - 0 out of 4 jobs completed
2020-04-21 17:29:12,022 - Waiting for message
2020-04-21 17:29:18,360 - Received message: {'job_id': '9459730', 'host_name': 'node514', 'ip_address': '172.18.3.108', 'command': 'heart_beat', 'data': {'memory': 24.7265625, 'cpu_load': (0.0, True), 'log_file': "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/runpy.py:125: RuntimeWarning: 'gridmap.runner' found in sys.modules after import of package 'gridmap', but prior to execution of 'gridmap.runner'; this may result in unpredictable behaviour\n warn(RuntimeWarning(msg))\n2020-04-21 17:29:18,265 - __main__ - INFO - Appended /ebio/abt3_projects/software/dev/gridmap/examples to PYTHONPATH\n2020-04-21 17:29:18,265 - __main__ - INFO - Starting heart beat\n2020-04-21 17:29:18,270 - __main__ - INFO - Waiting 1 seconds before fetching input\n"}}
2020-04-21 17:29:18,361 - Sending reply: all good
2020-04-21 17:29:18,361 - 0 out of 4 jobs completed
2020-04-21 17:29:18,361 - Waiting for message
2020-04-21 17:29:18,362 - Received message: {'job_id': '9459729', 'host_name': 'node514', 'ip_address': '172.18.3.108', 'command': 'heart_beat', 'data': {'memory': 24.81640625, 'cpu_load': (0.0, True), 'log_file': "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/runpy.py:125: RuntimeWarning: 'gridmap.runner' found in sys.modules after import of package 'gridmap', but prior to execution of 'gridmap.runner'; this may result in unpredictable behaviour\n warn(RuntimeWarning(msg))\n2020-04-21 17:29:18,265 - __main__ - INFO - Appended /ebio/abt3_projects/software/dev/gridmap/examples to PYTHONPATH\n2020-04-21 17:29:18,265 - __main__ - INFO - Starting heart beat\n2020-04-21 17:29:18,270 - __main__ - INFO - Waiting 4 seconds before fetching input\n"}}
2020-04-21 17:29:18,362 - Sending reply: all good
2020-04-21 17:29:18,362 - 0 out of 4 jobs completed
2020-04-21 17:29:18,362 - Waiting for message
2020-04-21 17:29:18,378 - Received message: {'job_id': '9459728', 'host_name': 'node522', 'ip_address': '172.18.3.11', 'command': 'heart_beat', 'data': {'memory': 24.75390625, 'cpu_load': (0.0, True), 'log_file': "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/runpy.py:125: RuntimeWarning: 'gridmap.runner' found in sys.modules after import of package 'gridmap', but prior to execution of 'gridmap.runner'; this may result in unpredictable behaviour\n warn(RuntimeWarning(msg))\n2020-04-21 17:29:18,267 - __main__ - INFO - Appended /ebio/abt3_projects/software/dev/gridmap/examples to PYTHONPATH\n2020-04-21 17:29:18,267 - __main__ - INFO - Starting heart beat\n2020-04-21 17:29:18,272 - __main__ - INFO - Waiting 1 seconds before fetching input\n"}}
2020-04-21 17:29:18,378 - Sending reply: all good
2020-04-21 17:29:18,378 - 0 out of 4 jobs completed
2020-04-21 17:29:18,378 - Waiting for message
2020-04-21 17:29:18,389 - Received message: {'job_id': '9459727', 'host_name': 'node522', 'ip_address': '172.18.3.11', 'command': 'heart_beat', 'data': {'memory': 24.74609375, 'cpu_load': (0.0, True), 'log_file': "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/runpy.py:125: RuntimeWarning: 'gridmap.runner' found in sys.modules after import of package 'gridmap', but prior to execution of 'gridmap.runner'; this may result in unpredictable behaviour\n warn(RuntimeWarning(msg))\n2020-04-21 17:29:18,267 - __main__ - INFO - Appended /ebio/abt3_projects/software/dev/gridmap/examples to PYTHONPATH\n2020-04-21 17:29:18,267 - __main__ - INFO - Starting heart beat\n2020-04-21 17:29:18,272 - __main__ - INFO - Waiting 3 seconds before fetching input\n"}}
2020-04-21 17:29:18,389 - Sending reply: all good
2020-04-21 17:29:18,389 - 0 out of 4 jobs completed
2020-04-21 17:29:18,389 - Waiting for message
2020-04-21 17:29:19,278 - Received message: {'job_id': '9459730', 'host_name': 'node514', 'ip_address': '172.18.3.108', 'command': 'fetch_input', 'data': None}
2020-04-21 17:29:19,278 - Received input request from 9459730
2020-04-21 17:29:19,278 - Sending reply: <gridmap.job.Job object at 0x7f627a0c0d08>
2020-04-21 17:29:19,281 - 0 out of 4 jobs completed
2020-04-21 17:29:19,281 - Waiting for message
2020-04-21 17:29:19,304 - Received message: {'job_id': '9459728', 'host_name': 'node522', 'ip_address': '172.18.3.11', 'command': 'fetch_input', 'data': None}
2020-04-21 17:29:19,304 - Received input request from 9459728
2020-04-21 17:29:19,305 - Sending reply: <gridmap.job.Job object at 0x7f627a0c0ae8>
2020-04-21 17:29:19,306 - 0 out of 4 jobs completed
2020-04-21 17:29:19,307 - Waiting for message
2020-04-21 17:29:19,317 - Received message: {'job_id': '9459730', 'host_name': 'node514', 'ip_address': '172.18.3.108', 'command': 'store_output', 'data': (AttributeError("'Job' object has no attribute '_f'",), 'Traceback (most recent call last):\n File "/ebio/abt3_projects/software/dev/gridmap/gridmap/runner.py", line 204, in _run_job\n job = _send_zmq_msg(job_id, "fetch_input", None, address)\n File "/ebio/abt3_projects/software/dev/gridmap/gridmap/runner.py", line 87, in _send_zmq_msg\n msg = zloads(zsocket.recv())\n File "/ebio/abt3_projects/software/dev/gridmap/gridmap/data.py", line 64, in zloads\n return cloudpickle.loads(bz2.decompress(pickled_data))\nAttributeError: \'Job\' object has no attribute \'_f\'\n')}
2020-04-21 17:29:19,317 - Received exception from 9459730
2020-04-21 17:29:19,317 - Sending reply: thanks
2020-04-21 17:29:19,317 - 0 out of 4 jobs completed
2020-04-21 17:29:19,317 - Waiting for message
2020-04-21 17:29:19,344 - Received message: {'job_id': '9459728', 'host_name': 'node522', 'ip_address': '172.18.3.11', 'command': 'store_output', 'data': (AttributeError("'Job' object has no attribute '_f'",), 'Traceback (most recent call last):\n File "/ebio/abt3_projects/software/dev/gridmap/gridmap/runner.py", line 204, in _run_job\n job = _send_zmq_msg(job_id, "fetch_input", None, address)\n File "/ebio/abt3_projects/software/dev/gridmap/gridmap/runner.py", line 87, in _send_zmq_msg\n msg = zloads(zsocket.recv())\n File "/ebio/abt3_projects/software/dev/gridmap/gridmap/data.py", line 64, in zloads\n return cloudpickle.loads(bz2.decompress(pickled_data))\nAttributeError: \'Job\' object has no attribute \'_f\'\n')}
2020-04-21 17:29:19,344 - Received exception from 9459728
2020-04-21 17:29:19,344 - Sending reply: thanks
2020-04-21 17:29:19,344 - 0 out of 4 jobs completed
2020-04-21 17:29:19,344 - Waiting for message
2020-04-21 17:29:21,288 - Received message: {'job_id': '9459727', 'host_name': 'node522', 'ip_address': '172.18.3.11', 'command': 'fetch_input', 'data': None}
2020-04-21 17:29:21,289 - Received input request from 9459727
2020-04-21 17:29:21,289 - Sending reply: <gridmap.job.Job object at 0x7f627a0c09d8>
2020-04-21 17:29:21,291 - 0 out of 4 jobs completed
2020-04-21 17:29:21,291 - Waiting for message
2020-04-21 17:29:21,313 - Received message: {'job_id': '9459727', 'host_name': 'node522', 'ip_address': '172.18.3.11', 'command': 'store_output', 'data': (AttributeError("'Job' object has no attribute '_f'",), 'Traceback (most recent call last):\n File "/ebio/abt3_projects/software/dev/gridmap/gridmap/runner.py", line 204, in _run_job\n job = _send_zmq_msg(job_id, "fetch_input", None, address)\n File "/ebio/abt3_projects/software/dev/gridmap/gridmap/runner.py", line 87, in _send_zmq_msg\n msg = zloads(zsocket.recv())\n File "/ebio/abt3_projects/software/dev/gridmap/gridmap/data.py", line 64, in zloads\n return cloudpickle.loads(bz2.decompress(pickled_data))\nAttributeError: \'Job\' object has no attribute \'_f\'\n')}
2020-04-21 17:29:21,313 - Received exception from 9459727
2020-04-21 17:29:21,314 - Sending reply: thanks
2020-04-21 17:29:21,314 - 0 out of 4 jobs completed
2020-04-21 17:29:21,314 - Waiting for message
2020-04-21 17:29:22,278 - Received message: {'job_id': '9459729', 'host_name': 'node514', 'ip_address': '172.18.3.108', 'command': 'fetch_input', 'data': None}
2020-04-21 17:29:22,278 - Received input request from 9459729
2020-04-21 17:29:22,278 - Sending reply: <gridmap.job.Job object at 0x7f627a0c0bf8>
2020-04-21 17:29:22,280 - 0 out of 4 jobs completed
2020-04-21 17:29:22,280 - Waiting for message
2020-04-21 17:29:22,285 - Received message: {'job_id': '9459729', 'host_name': 'node514', 'ip_address': '172.18.3.108', 'command': 'store_output', 'data': (AttributeError("'Job' object has no attribute '_f'",), 'Traceback (most recent call last):\n File "/ebio/abt3_projects/software/dev/gridmap/gridmap/runner.py", line 204, in _run_job\n job = _send_zmq_msg(job_id, "fetch_input", None, address)\n File "/ebio/abt3_projects/software/dev/gridmap/gridmap/runner.py", line 87, in _send_zmq_msg\n msg = zloads(zsocket.recv())\n File "/ebio/abt3_projects/software/dev/gridmap/gridmap/data.py", line 64, in zloads\n return cloudpickle.loads(bz2.decompress(pickled_data))\nAttributeError: \'Job\' object has no attribute \'_f\'\n')}
2020-04-21 17:29:22,285 - Received exception from 9459729
2020-04-21 17:29:22,285 - Sending reply: thanks
2020-04-21 17:29:22,285 - 0 out of 4 jobs completed
2020-04-21 17:29:22,286 - Waiting for message
2020-04-21 17:29:27,035 - Connecting to JobMonitor (tcp://172.18.3.170:41971)
2020-04-21 17:29:27,035 - Sending message: {'job_id': -1, 'host_name': 'rick', 'ip_address': '172.18.3.170', 'command': 'heart_beat', 'data': {}}
2020-04-21 17:29:27,036 - Received message: {'job_id': -1, 'host_name': 'rick', 'ip_address': '172.18.3.170', 'command': 'heart_beat', 'data': {}}
2020-04-21 17:29:27,036 - Checking if jobs are alive
2020-04-21 17:29:27,078 - findfont: Matching sans\-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0.
2020-04-21 17:29:27,079 - findfont: score(<Font 'STIXGeneral' (STIXGeneral.ttf) normal normal regular normal>) = 10.05
2020-04-21 17:29:27,079 - findfont: score(<Font 'STIXSizeFourSym' (STIXSizFourSymReg.ttf) normal normal regular normal>) = 10.05
2020-04-21 17:29:27,079 - findfont: score(<Font 'DejaVu Sans Mono' (DejaVuSansMono-BoldOblique.ttf) oblique normal bold normal>) = 11.335
2020-04-21 17:29:27,079 - findfont: score(<Font 'cmb10' (cmb10.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,079 - findfont: score(<Font 'STIXSizeThreeSym' (STIXSizThreeSymBol.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,079 - findfont: score(<Font 'STIXGeneral' (STIXGeneralBol.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,079 - findfont: score(<Font 'cmmi10' (cmmi10.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,079 - findfont: score(<Font 'STIXNonUnicode' (STIXNonUniBol.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,079 - findfont: score(<Font 'cmex10' (cmex10.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,079 - findfont: score(<Font 'DejaVu Sans' (DejaVuSans-Bold.ttf) normal normal bold normal>) = 0.33499999999999996
2020-04-21 17:29:27,079 - findfont: score(<Font 'DejaVu Sans' (DejaVuSans.ttf) normal normal 400 normal>) = 0.05
2020-04-21 17:29:27,080 - findfont: score(<Font 'STIXSizeTwoSym' (STIXSizTwoSymBol.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,080 - findfont: score(<Font 'STIXSizeFiveSym' (STIXSizFiveSymReg.ttf) normal normal regular normal>) = 10.05
2020-04-21 17:29:27,080 - findfont: score(<Font 'cmss10' (cmss10.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,080 - findfont: score(<Font 'DejaVu Serif Display' (DejaVuSerifDisplay.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,080 - findfont: score(<Font 'STIXSizeOneSym' (STIXSizOneSymReg.ttf) normal normal regular normal>) = 10.05
2020-04-21 17:29:27,080 - findfont: score(<Font 'STIXGeneral' (STIXGeneralItalic.ttf) italic normal 400 normal>) = 11.05
2020-04-21 17:29:27,080 - findfont: score(<Font 'DejaVu Serif' (DejaVuSerif-BoldItalic.ttf) italic normal bold normal>) = 11.335
2020-04-21 17:29:27,080 - findfont: score(<Font 'cmtt10' (cmtt10.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,080 - findfont: score(<Font 'DejaVu Serif' (DejaVuSerif-Italic.ttf) italic normal 400 normal>) = 11.05
2020-04-21 17:29:27,080 - findfont: score(<Font 'DejaVu Sans' (DejaVuSans-Oblique.ttf) oblique normal 400 normal>) = 1.05
2020-04-21 17:29:27,080 - findfont: score(<Font 'DejaVu Serif' (DejaVuSerif.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,080 - findfont: score(<Font 'cmsy10' (cmsy10.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,080 - findfont: score(<Font 'DejaVu Sans Mono' (DejaVuSansMono-Bold.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,080 - findfont: score(<Font 'STIXNonUnicode' (STIXNonUni.ttf) normal normal regular normal>) = 10.05
2020-04-21 17:29:27,081 - findfont: score(<Font 'DejaVu Sans Mono' (DejaVuSansMono.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,081 - findfont: score(<Font 'STIXNonUnicode' (STIXNonUniBolIta.ttf) italic normal bold normal>) = 11.335
2020-04-21 17:29:27,081 - findfont: score(<Font 'DejaVu Serif' (DejaVuSerif-Bold.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,081 - findfont: score(<Font 'STIXSizeOneSym' (STIXSizOneSymBol.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,081 - findfont: score(<Font 'STIXSizeTwoSym' (STIXSizTwoSymReg.ttf) normal normal regular normal>) = 10.05
2020-04-21 17:29:27,081 - findfont: score(<Font 'cmr10' (cmr10.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,081 - findfont: score(<Font 'STIXGeneral' (STIXGeneralBolIta.ttf) italic normal bold normal>) = 11.335
2020-04-21 17:29:27,081 - findfont: score(<Font 'DejaVu Sans' (DejaVuSans-BoldOblique.ttf) oblique normal bold normal>) = 1.335
2020-04-21 17:29:27,081 - findfont: score(<Font 'DejaVu Sans Display' (DejaVuSansDisplay.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,081 - findfont: score(<Font 'STIXSizeFourSym' (STIXSizFourSymBol.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,081 - findfont: score(<Font 'STIXNonUnicode' (STIXNonUniIta.ttf) italic normal 400 normal>) = 11.05
2020-04-21 17:29:27,081 - findfont: score(<Font 'STIXSizeThreeSym' (STIXSizThreeSymReg.ttf) normal normal regular normal>) = 10.05
2020-04-21 17:29:27,081 - findfont: score(<Font 'DejaVu Sans Mono' (DejaVuSansMono-Oblique.ttf) oblique normal 400 normal>) = 11.05
2020-04-21 17:29:27,082 - findfont: score(<Font 'wasy10' (wasy10.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,082 - findfont: score(<Font 'Liberation Serif' (LiberationSerif-Bold.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,082 - findfont: score(<Font 'DejaVu Serif' (DejaVuSerifCondensed.ttf) normal normal 400 condensed>) = 10.25
2020-04-21 17:29:27,082 - findfont: score(<Font 'Liberation Mono' (LiberationMono-BoldItalic.ttf) italic normal bold normal>) = 11.335
2020-04-21 17:29:27,082 - findfont: score(<Font 'Liberation Sans Narrow' (LiberationSansNarrow-Regular.ttf) normal normal 400 condensed>) = 10.25
2020-04-21 17:29:27,082 - findfont: score(<Font 'Lato' (Lato-Light.ttf) normal normal light normal>) = 10.24
2020-04-21 17:29:27,082 - findfont: score(<Font 'Droid Sans Fallback' (DroidSansFallbackFull.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,082 - findfont: score(<Font 'Lato' (Lato-Black.ttf) normal normal black normal>) = 10.525
2020-04-21 17:29:27,082 - findfont: score(<Font 'Liberation Sans' (LiberationSans-Italic.ttf) italic normal 400 normal>) = 11.05
2020-04-21 17:29:27,082 - findfont: score(<Font 'Bitstream Vera Sans' (VeraBd.ttf) normal normal bold normal>) = 1.2440909090909091
2020-04-21 17:29:27,082 - findfont: score(<Font 'FreeSerif' (FreeSerifItalic.ttf) italic normal 400 normal>) = 11.05
2020-04-21 17:29:27,082 - findfont: score(<Font 'Bitstream Vera Sans' (VeraIt.ttf) oblique normal 400 normal>) = 1.9590909090909092
2020-04-21 17:29:27,083 - findfont: score(<Font 'Liberation Mono' (LiberationMono-Bold.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,083 - findfont: score(<Font 'Lato' (Lato-ThinItalic.ttf) italic normal 400 normal>) = 11.05
2020-04-21 17:29:27,083 - findfont: score(<Font 'DejaVu Sans' (DejaVuSansCondensed.ttf) normal normal 400 condensed>) = 0.25
2020-04-21 17:29:27,083 - findfont: score(<Font 'Lato' (Lato-LightItalic.ttf) italic normal light normal>) = 11.24
2020-04-21 17:29:27,083 - findfont: score(<Font 'Liberation Serif' (LiberationSerif-Regular.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,083 - findfont: score(<Font 'Lato' (Lato-MediumItalic.ttf) italic normal medium normal>) = 11.145
2020-04-21 17:29:27,083 - findfont: score(<Font 'Liberation Serif' (LiberationSerif-BoldItalic.ttf) italic normal bold normal>) = 11.335
2020-04-21 17:29:27,083 - findfont: score(<Font 'stmary10' (stmary10.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,083 - findfont: score(<Font 'DejaVu Sans' (DejaVuSansCondensed-Oblique.ttf) oblique normal 400 condensed>) = 1.25
2020-04-21 17:29:27,083 - findfont: score(<Font 'Liberation Sans' (LiberationSans-Regular.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,083 - findfont: score(<Font 'Bitstream Vera Serif' (VeraSe.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,083 - findfont: score(<Font 'Liberation Serif' (LiberationSerif-Italic.ttf) italic normal 400 normal>) = 11.05
2020-04-21 17:29:27,083 - findfont: score(<Font 'eufm10' (eufm10.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,083 - findfont: score(<Font 'Bitstream Vera Sans' (Vera.ttf) normal normal 400 normal>) = 0.9590909090909092
2020-04-21 17:29:27,084 - findfont: score(<Font 'Liberation Sans' (LiberationSans-Bold.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,084 - findfont: score(<Font 'Lato' (Lato-BoldItalic.ttf) italic normal bold normal>) = 11.335
2020-04-21 17:29:27,084 - findfont: score(<Font 'DejaVu Sans' (DejaVuSansCondensed-Bold.ttf) normal normal bold condensed>) = 0.5349999999999999
2020-04-21 17:29:27,084 - findfont: score(<Font 'FreeSans' (FreeSans.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,084 - findfont: score(<Font 'Lato' (Lato-Medium.ttf) normal normal medium normal>) = 10.145
2020-04-21 17:29:27,084 - findfont: score(<Font 'DejaVu Serif' (DejaVuSerifCondensed-Italic.ttf) italic normal 400 condensed>) = 11.25
2020-04-21 17:29:27,084 - findfont: score(<Font 'DejaVu Sans' (DejaVuSans-ExtraLight.ttf) normal normal light normal>) = 0.24
2020-04-21 17:29:27,084 - findfont: score(<Font 'Lato' (Lato-SemiboldItalic.ttf) italic normal semibold normal>) = 11.24
2020-04-21 17:29:27,084 - findfont: score(<Font 'Liberation Sans Narrow' (LiberationSansNarrow-Italic.ttf) italic normal 400 condensed>) = 11.25
2020-04-21 17:29:27,084 - findfont: score(<Font 'Bitstream Vera Sans Mono' (VeraMoBd.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,084 - findfont: score(<Font 'DejaVu Serif' (DejaVuSerifCondensed-BoldItalic.ttf) italic normal bold condensed>) = 11.535
2020-04-21 17:29:27,084 - findfont: score(<Font 'Liberation Mono' (LiberationMono-Italic.ttf) italic normal 400 normal>) = 11.05
2020-04-21 17:29:27,084 - findfont: score(<Font 'Lato' (Lato-HeavyItalic.ttf) italic normal heavy normal>) = 11.43
2020-04-21 17:29:27,085 - findfont: score(<Font 'Noto Mono' (NotoMono-Regular.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,085 - findfont: score(<Font 'FreeSans' (FreeSansOblique.ttf) oblique normal 400 normal>) = 11.05
2020-04-21 17:29:27,085 - findfont: score(<Font 'FreeMono' (FreeMonoBold.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,085 - findfont: score(<Font 'Liberation Sans' (LiberationSans-BoldItalic.ttf) italic normal bold normal>) = 11.335
2020-04-21 17:29:27,085 - findfont: score(<Font 'FreeMono' (FreeMonoBoldOblique.ttf) oblique normal bold normal>) = 11.335
2020-04-21 17:29:27,085 - findfont: score(<Font 'msbm10' (msbm10.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,085 - findfont: score(<Font 'Liberation Sans Narrow' (LiberationSansNarrow-Bold.ttf) normal normal bold condensed>) = 10.535
2020-04-21 17:29:27,085 - findfont: score(<Font 'Lato' (Lato-BlackItalic.ttf) italic normal black normal>) = 11.525
2020-04-21 17:29:27,085 - findfont: score(<Font 'FreeSerif' (FreeSerifBold.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,085 - findfont: score(<Font 'Liberation Mono' (LiberationMono-Regular.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,085 - findfont: score(<Font 'Bitstream Vera Sans Mono' (VeraMoIt.ttf) oblique normal 400 normal>) = 11.05
2020-04-21 17:29:27,085 - findfont: score(<Font 'Lato' (Lato-HairlineItalic.ttf) italic normal 400 normal>) = 11.05
2020-04-21 17:29:27,085 - findfont: score(<Font 'FreeSerif' (FreeSerifBoldItalic.ttf) italic normal bold normal>) = 11.335
2020-04-21 17:29:27,086 - findfont: score(<Font 'DejaVu Serif' (DejaVuSerifCondensed-Bold.ttf) normal normal bold condensed>) = 10.535
2020-04-21 17:29:27,086 - findfont: score(<Font 'Liberation Sans Narrow' (LiberationSansNarrow-BoldItalic.ttf) italic normal bold condensed>) = 11.535
2020-04-21 17:29:27,086 - findfont: score(<Font 'DejaVu Sans' (DejaVuSansCondensed-BoldOblique.ttf) oblique normal bold condensed>) = 1.535
2020-04-21 17:29:27,086 - findfont: score(<Font 'msam10' (msam10.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,086 - findfont: score(<Font 'Bitstream Vera Sans Mono' (VeraMoBI.ttf) oblique normal bold normal>) = 11.335
2020-04-21 17:29:27,086 - findfont: score(<Font 'Bitstream Vera Sans Mono' (VeraMono.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,086 - findfont: score(<Font 'Lato' (Lato-Italic.ttf) italic normal 400 normal>) = 11.05
2020-04-21 17:29:27,086 - findfont: score(<Font 'Lato' (Lato-Thin.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,086 - findfont: score(<Font 'FreeSerif' (FreeSerif.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,086 - findfont: score(<Font 'Bitstream Vera Sans' (VeraBI.ttf) oblique normal bold normal>) = 2.244090909090909
2020-04-21 17:29:27,086 - findfont: score(<Font 'Lato' (Lato-Hairline.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,086 - findfont: score(<Font 'FreeMono' (FreeMonoOblique.ttf) oblique normal 400 normal>) = 11.05
2020-04-21 17:29:27,086 - findfont: score(<Font 'Lato' (Lato-Semibold.ttf) normal normal semibold normal>) = 10.24
2020-04-21 17:29:27,087 - findfont: score(<Font 'FreeSans' (FreeSansBold.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,087 - findfont: score(<Font 'Lato' (Lato-Bold.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,087 - findfont: score(<Font 'FreeMono' (FreeMono.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,087 - findfont: score(<Font 'esint10' (esint10.ttf) normal normal 400 normal>) = 10.05
2020-04-21 17:29:27,087 - findfont: score(<Font 'FreeSans' (FreeSansBoldOblique.ttf) oblique normal bold normal>) = 11.335
2020-04-21 17:29:27,087 - findfont: score(<Font 'DejaVu Math TeX Gyre' (DejaVuMathTeXGyre.ttf) normal normal regular normal>) = 10.05
2020-04-21 17:29:27,087 - findfont: score(<Font 'Bitstream Vera Serif' (VeraSeBd.ttf) normal normal bold normal>) = 10.335
2020-04-21 17:29:27,087 - findfont: score(<Font 'Lato' (Lato-Heavy.ttf) normal normal heavy normal>) = 10.43
2020-04-21 17:29:27,087 - findfont: score(<Font 'Lato' (Lato-Regular.ttf) normal normal regular normal>) = 10.05
2020-04-21 17:29:27,087 - findfont: score(<Font 'rsfs10' (rsfs10.ttf) italic normal 400 normal>) = 11.05
2020-04-21 17:29:27,087 - findfont: Matching sans\-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0 to DejaVu Sans ('/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf') with score of 0.050000.
2020-04-21 17:29:27,376 - Encountered ConnectionRefusedError, so killing all jobs.
Traceback (most recent call last):
File "./examples/map_reduce.py", line 115, in <module>
main(args)
File "./examples/map_reduce.py", line 105, in main
max_processes=4, queue='long.q')
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/site-packages/gridmap/job.py", line 1009, in grid_map
require_cluster=require_cluster)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/site-packages/gridmap/job.py", line 896, in process_jobs
monitor.check(sid, jobs)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/site-packages/gridmap/job.py", line 452, in check
self.check_if_alive()
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/site-packages/gridmap/job.py", line 499, in check_if_alive
send_error_mail(job)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/site-packages/gridmap/job.py", line 674, in send_error_mail
_send_mail(subject, body_text, attachments)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/site-packages/gridmap/job.py", line 564, in _send_mail
s = smtplib.SMTP(SMTP_SERVER)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/smtplib.py", line 336, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/smtplib.py", line 307, in _get_socket
self.source_address)
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/socket.py", line 724, in create_connection
raise err
File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/quast_par/lib/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
mulhod commented
This is interesting. You might want to take a look at this commit here that was merged not too long ago. In that commit, the _f attribute was removed from Job. Is it possible the version of gridmap you have installed does not have this change in the other host? (I'm not sure what your setup is, but maybe you have 2+ environments and they are out of sync?)
Also, I wonder why it appears to still be trying to send email? The error occurs before then, but it looks like it is perhaps still using the default value of SEND_ERROR_MAIL.
nick-youngblut commented
I created a new conda env and installed the latest commit of gridmap. gridmap now working correctly. Thanks!
I also like the Could not import matplotlib. No plots will be created in debug emails. feature.