How are errors in concurrent function handled?
Closed this issue · 12 comments
Great library! Ideally I'd like a way for an error to stop the whole program. is that possible?
That's definitely possible, exceptions that are raised in @concurrent functions get raised in their calling context when synchronization happens. Do you have a program where exceptions aren't being raised from an @concurrent function?
Thanks! I didn't even think to try it ... I'll report back if it doesn't work.
Wow it does work. Thanks.
Awesome, glad it works. Let me know if you find something else that's not working!
I don't suppose there is any good way to make the entire program exit before synchronization?
I just pushed a change that should make Ctrl + C work, so give that a shot. If you're on Windows though, Ctrl + Pause Break should work with the last version as well.
There's a sort of bug in multiprocessing.Pool where if you don't specify a time out to their blocking call, KeyboardInterrupts won't get handled. My solution is to just pass like a large timeout to get().
Thanks! I made that change on my install. .4.1. But it still doesn't seem to respect keyboard interrupt. Just keeps going.
I just tried this change out in 4.1 on Windows and Linux and it did exit on keyboard interrupt. Are you using OSX by chance, which I haven't been able to test? Otherwise could you verify that a short example also doesn't work?
from deco import *
import time
sleep = concurrent(lambda t: time.sleep(t))
if __name__ == "__main__":
sleep(10).get()If interrupting this one does work, probably there is something going on with your specific program. If it doesn't work, there is probably an issue with the way you applied the change, that maybe it's not actually getting applied.
I am on osx for development, deploying to linux for production.
But your last idea was actually right. I was editing the wrong code. It seems to work! Thanks.
Would this make it into pip soon?
Hmm, it's odd. If I hit ctrl+c this fix works and everything stops.
But if I raise KeyboardInterrupt from inside a job it still doesn't stop the main process. I wonder what the difference is.
The difference is when you cause a KeyboardInterrupt in the main thread vs. in the child thread. The main thread is waiting for all these child processes to wrap up their work, so when you interrupt the main thread it can exit like right away. But when you throw an exception in a child process, the main thread needs to get to that child. So it waits for the first child's result, then the second's etc until it tries to wait on the child that threw an exception. That child's result has probably been ready for a while, but it hasn't been waited on until now. At this moment the exception gets thrown in the main thread.
Making it so that the main thread handles the child results as they come back is kind of complicated I think and probably not something I have time to address in the near future.
Yes, that makes perfect sense. I understand that would be hard to change. I'll see what I can figure out as a work around.