dubovikmaster/parallelbar

AssertionError in 2.3.1 when context='forkserver' / 'spawn' in Linux

Closed this issue · 3 comments

jonnyyu commented

Hi @dubovikmaster,

Our system updated to the latest 2.3.1, but I get the following assertion on Linux.

AssertionError: daemonic processes are not allowed to have children

Here's the exception stacktrace (in child process):

Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/parallelbar/__init__.py", line 1, in <module>
    from .parallelbar import progress_map, progress_imap, progress_imapu, progress_starmap
  File "/home/ubuntu/.local/lib/python3.8/site-packages/parallelbar/parallelbar.py", line 15, in <module>
    from .wrappers import (
  File "/home/ubuntu/.local/lib/python3.8/site-packages/parallelbar/wrappers.py", line 10, in <module>
    from ._worker_queue import _WORKER_QUEUE
  File "/home/ubuntu/.local/lib/python3.8/site-packages/parallelbar/_worker_queue.py", line 7, in <module>
    _WORKER_QUEUE = mp.Manager().Queue()
  File "/usr/lib/python3.8/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 579, in start
    self._process.start()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 118, in start
    assert not _current_process._config.get('daemon'), \

From the stack it seems the assertion is because the child process loads the parallelbar module too, and the new version now always creates a Queue(). Is there anyway to avoid creating Queue() object in child process?

jonnyyu commented

This issue doesn't occur when using context=fork, but in my case I need use forkserver otherwise fsspec will hang

dubovikmaster commented

Hello! Can you give me a code example where you use parallelbar?

jonnyyu commented

Hi @dubovikmaster, I just tried the 2.4 and the issue is fixed now. Thanks for your quick help!

  • 🎉1