Threading issue
olivierlabayle opened this issue · 3 comments
Hi there,
I think a threading issue was introduced in 0.15. The following works on 0.14 but raises @threads :static cannot be used concurrently or nested
in 0.15:
using MLJ
using EvoTrees
using MLJLinearModels
X, y = make_regression()
model = Stack(
metalearner = LinearRegressor(),
resampling = CV(nfolds=2),
tree = EvoTreeRegressor()
)
mach = machine(model, X, y)
fit!(mach)
Not sure if it is platform specific, I am on MAC M1 with MLJ v0.19.2.
The move to the :static
threads scheduling was made in the context of several discussions arising around some race conditions:
- https://discourse.julialang.org/t/psa-thread-local-state-is-no-longer-recommended-common-misconceptions-about-threadid-and-nthreads/101274/45
- https://julialang.org/blog/2023/07/PSA-dont-use-threadid/
But the caveat with :static
is what you just experienced, where MLJ apparently uses threading for some of its routines.
I'll have a second look at the above mentionned caveats as I don't think the code was subject to those. If such is the case, then I should be ably to reinstate the dynamic
mode which allows for nested parallelism.
@olivierlabayle Could you give a try at https://github.com/Evovest/EvoTrees.jl/tree/threading?
I've removed the :static
to use :dynamic
threading. The above MLJ test ran fine on my end (Windows).
I should then be able to make a fix release by tomorrow.
@olivierlabayle Could you give a try at https://github.com/Evovest/EvoTrees.jl/tree/threading? I've removed the
:static
to use:dynamic
threading. The above MLJ test ran fine on my end (Windows). I should then be able to make a fix release by tomorrow.
Since you've closed this, I've just tried the new 0.16 which works just fine. Thank you for solving it so fast! :)