tro3/ThreadPools.jl

qmap hangs forever when using do notation

Closed this issue · 14 comments

samuela commented

When you accidentally forget the input parameter and write

julia> ThreadPools.qmap(1:10) do
           0
       end
...

It just hangs forever. I would hope it could produce an error instead when the input arguments are incorrect in some way.

samuela commented

In contrast to

julia> ThreadPools.qmap(1:10) do _
           0
       end

which does work as expected.

samuela commented

Also seems to happen when the function signature doesn't exactly match in other ways:

julia> ThreadPools.qmap(1:3) do (a, b)
           0
       end
...

also hangs...

tro3 commented

I think this one is Julia's issue with exception handling in threads, but I'll take a look.

tro3 commented

Addressed in v1.1.2, which will propagate in the next hour or so. I put an assertion ahead of the threading to handle this.

  • 👍1
danielwe commented

I'm on ThreadPools 1.1.3 and Julia 1.5.3 and still have this problem. More precisely, any exception raised within a job that was assigned using dynamic scheduling causes the code to hang forever. When using static scheduling and spawning, exceptions interrupt execution as expected, complete with nested exceptions and stacktraces.

julia> qforeach(_ -> error(), 1:Threads.nthreads())  # does not work
<silence, press Ctrl-C to get back to the prompt>

julia> @qthreads for _ in 1:Threads.nthreads(); error(); end  # does not work
<silence, as above>

julia> Threads.@threads for _ in 1:Threads.nthreads(); error(); end  # works
ERROR: TaskFailedException:

Stacktrace:
 [1] error() at ./error.jl:42
 [2] [...]
Stacktrace:
 [1] wait at ./task.jl:267 [inlined]
 [2] [...]

julia> @bthreads for _ in 1:Threads.nthreads(); error(); end  # works
ERROR: TaskFailedException:

Stacktrace:
 [1] error() at ./error.jl:42
 [2] [...]
Stacktrace:
 [1] wait at ./task.jl:267 [inlined]
 [2] [...]

julia> fetch(@tspawnat 2 error())
ERROR: TaskFailedException:

Stacktrace:
 [1] error() at ./error.jl:42
 [2] [...]
Stacktrace:
 [1] wait at ./task.jl:267 [inlined]
 [2] [...]
tro3 commented

Yes, the core "Exception during thread" issue is Julia's. I haven't seen any updates in 1.5 that address this, but I'll query to make sure. If they have added some hooks, I'll fold into ThreadPools. Thx

danielwe commented

Great! Multithreading in Julia is clearly immature and a work in progress, but Threads.@threads and Threads.@spawn actually seem to handle exceptions sensibly on my machine. Not familiar with the lower-level hooks ThreadPools needs, though.

  • 👍1
tro3 commented

!! Ok, if you are getting decent behavior out of @threads, then I have some work to do to try to match it (the point with Threadpools.@?threads is to match @threads). I'll take a look.

tro3 commented

Okay, looks like it is only a @qthreads issue, as @bthreads does now raise the Exception. (I don't think it did in earlier version of Julia, but I could be wrong). I'll dig and see if I can improve exception handling here - it has been a pain for me as well. :-)

tro3 commented

Sigh. Nope. This issue is dependent on JuliaLang/julia#32677, which is still pending.

danielwe commented

Thanks for investigating!

  • 👍1
tro3 commented

Aha! In looking at the Julia issue and wrapping my head around the async scheduling a bit more, I find that this is not dependent on 32677 after all. It is a subtle issue with the QueuePool handler. Will submit an update shortly.

tro3 commented

Ok, problem is fixed for both of the above MWE's, and v1.1.4 is making its way through the registration process. Sorry for the delay - my bad for allowing the red herring to deter me.

  • 🎉2
danielwe commented

Awesome! Thanks, and thanks so much for this package.

  • 👍1