Raising an exception that is unable to be unpickled causes hang in ProcessPoolExecutor
GoogleCodeExporter opened this issue · 9 comments
What steps will reproduce the problem?
1. In the function submitted to a ProcessPoolExecutor, raise a custom exception
class that takes more than one argument to __init__.
What is the expected output? What do you see instead?
I expect a call to future.result() to not hang.
What version of the product are you using? On what operating system?
I'm using ver 2.1.6 on python 2.7 on Gentoo Linux.
Please provide any additional information below.
I have attached a patch to address the issue and a test case for it. Without
the patch, the new test case hangs. With the patch, it passes.
This is needed because of the issue raised in
http://bugs.python.org/issue1692335. An exception class that takes multiple
arguments to __init__ can be pickled but it raises a TypeError when being
unpickled:
In [1]: class MyError(Exception):
...: def __init__(self, arg1, arg2):
...: super(MyError, self).__init__(
...: 'arg1 = {}, arg2 = {}'.format(arg1, arg2))
...:
In [2]: import pickle
In [3]: p = pickle.dumps(MyError('arg1val', 'arg2val'))
In [4]: pickle.loads(p)
---------------------------------------------------------------------------
<snip>
TypeError: __init__() takes exactly 3 arguments (2 given)
So if a child process raises an exception like this, it gets pickled and put in
the result queue just fine. However, in _queue_management_worker, the call to
result_queue.get(block=True) will raise an uncaught TypeError when it tries to
unpickle the exception. So then the queue management just breaks.
My proposed patch attempts to catch this condition before putting the exception
in the result queue and create a new exception that will be able to be
unpickled but still contains information from the original exception.
Original issue reported on code.google.com by tbea...@gmail.com on 30 Sep 2014 at 2:23
Attachments:
I would need a test for this fix too.
When possible, you can workaround this by allowing this exception to have zero arguments (by default parameters). At least that fixed it for me until the patch is through.
The patch also contains a test, if I'm not mistaken ...
Hm, yes it does! How did I miss that.
I'm surprised that there is no fix for this in upstream code. Is it not a problem on Python 3?
It sure is. Frankly, I wasn't aware that this one here is the backports project, I'll file a bug for Python 3.3.
I'll wait until upstream developers comment on it until applying any fix.
This is the upstream bug: http://bugs.python.org/issue24900
Is this still relevant? If so, how do I reproduce it?