grnboost error
wangjiawen2013 opened this issue · 6 comments
Hello,
When I run "network = grnboost2(expression_data=ex_matrix,tf_names=tf_names)" according to the tutorial, the following error occured:
here is a snippet of the errors:
distributed.core - ERROR - Timed out trying to connect to 'tcp://127.0.0.1:33261' after 10 s: connect() didn't finish in time
tornado.util.TimeoutError: Timeout
OSError: Timed out trying to connect to 'tcp://127.0.0.1:33261' after 10 s: connect() didn't finish in time
tornado.application - ERROR - Exception in Future after timeout
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
Based on your pySCENIC issue, I think the Tornado error is not the root cause. Tornado probably times out because of a problem with the configuration of the numerical libraries on your system, OpenBLAS in this case.
Perhaps following StackOverflow question can help you resolve the matter:
https://stackoverflow.com/questions/51256738/multiple-instances-of-python-running-simultaneously-limited-to-35
PS: it really helps posting issues with more contextual information, like the other issue you filed, longer snippet from the error log, information on what kind of system you are running (operating system, specs etc.), and what Python environment you are using.
I still cannot run grnboost although I change OpenBLAS settings according to your indication, here are the detailed error message:
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
Traceback (most recent call last):
File "pipeline", line 18, in
adjacencies = grnboost2(ex_matrix, tf_names=tf_names, verbose=True)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/arboreto/algo.py", line 41, in grnboost2
early_stop_window_length=early_stop_window_length, limit=limit, seed=seed, verbose=verbose)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/arboreto/algo.py", line 109, in diy
client, shutdown_callback = _prepare_client(client_or_address)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/arboreto/algo.py", line 157, in _prepare_client
local_cluster = LocalCluster(diagnostics_port=None)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 141, in init
self.start(ip=ip, n_workers=n_workers)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 171, in start
self.sync(self._start, **kwargs)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 164, in sync
return sync(self.loop, func, *args, **kwargs)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/utils.py", line 277, in sync
six.reraise(*error[0])
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/utils.py", line 262, in f
result[0] = yield future
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 191, in _start
yield [self._start_worker(**self.worker_kwargs) for i in range(n_workers)]
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-35, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-56, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-62, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-13, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-2, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-38, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-27, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-16, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-41, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-5, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-30, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-60, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-19, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-8, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-44, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-47, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-11, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-14, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-1, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-3, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-53, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-17, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-42, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-6, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-31, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-20, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-59, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-63, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-23, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-48, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-15, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-40, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-54, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-18, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-7, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-64, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-21, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-10, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-46, started daemon)>
Exception ignored in: <bound method LocalCluster.del of LocalCluster('tcp://127.0.0.1:40281', workers=0, ncores=0)>
Traceback (most recent call last):
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 340, in del
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 291, in close
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/utils.py", line 425, in run_sync
File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/utils.py", line 272, in sync
tornado.util.TimeoutError: timed out after 20 s.
@wangjiawen2013 I have the same problem, Have you solve this problem? Thank you.
@vagabond12 solve it like this:
arboreto_with_multiprocessing.py
expr_mat.loom
allTFs_hg38.txt
--method grnboost2
--output adj.tsv
--num_workers 20
--seed 777
https://pyscenic.readthedocs.io/en/latest/faq.html
the latest pyscenic added a helper script arboreto_with_multiprocessing.py that runs the Arboreto GRN algorithms (GRNBoost2, GENIE3) without Dask for compatibility. Then you can import the results into jupyter notebook.
Thank you, I know this helper script. But I do not think this helper script runs the Arboreto GRN algorithms (GRNBoost2, GENIE3) without Dask for compatibility, Because it has the same warning of dask as pyscenic grn. If they all use dask, Why pyscenic grn always does not work? That is a question.
I think the warnings are irrevelant to dask. It's just on the configuration parameters (in .yaml format) or other things.