concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending
fenglonz opened this issue · 3 comments
Traceback (most recent call last):
File "./deepwalk", line 11, in
load_entry_point('deepwalk==1.0.3', 'console_scripts', 'deepwalk')()
File "/usr/local/python36/lib/python3.6/site-packages/deepwalk-1.0.3-py3.6.egg/deepwalk/main.py", line 162, in main
process(args)
File "/usr/local/python36/lib/python3.6/site-packages/deepwalk-1.0.3-py3.6.egg/deepwalk/main.py", line 83, in process
num_workers=args.workers)
File "/usr/local/python36/lib/python3.6/site-packages/deepwalk-1.0.3-py3.6.egg/deepwalk/walks.py", line 85, in write_walks_to_disk
for file_ in executor.map(_write_walks_to_disk, args_list):
File "/usr/local/python36/lib/python3.6/concurrent/futures/process.py", line 366, in _chain_from_iterable_of_lists
for element in iterable:
File "/usr/local/python36/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
yield fs.pop().result()
File "/usr/local/python36/lib/python3.6/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/local/python36/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
84 with ProcessPoolExecutor(max_workers=num_workers) as executor:
85 for file_ in executor.map(write_walks_to_disk, args_list):
86 files.append(file)
It fails at walks.py:85. How to fix?
What OS are you using? What is the exact command you ran? This question seems to be related: https://stackoverflow.com/questions/15900366/all-example-concurrent-futures-code-is-failing-with-brokenprocesspool
Yes, I too believe that it is related to the issue that you linked. The way that I run into this problem is when I try to run multiple workers, but can't keep all the walks in memory. So, when I try to run the following command on a large graph, it crashed with the original issue message above. I am using Debian Buster.
--number-walks 80 --representation-size 128 --walk-length 40 --window-size 10 --workers 20
Have you found a solution to this bug, or have any idea about how I can go about fixing it. I am not so familiar with the python ProcessPoolExecutor library, and as indicated by the article you linked the error messages aren't very helpful.