Enhanced new build_many to use on all platforms
Closed this issue · 10 comments
In #27490 I hacked together a replacement for the sage_setup.docbuild.build_many function, which implements (blocking) parallel map() of sorts, which solved some problems with using multiprocessing.Pool.map that stems from its use of threads to fork new processes.
That solved a problem that was specific to older versions of Cygwin. However, there is a similar problem, which affects all platforms, with PARI built with multi-threading support: #26608.
Although the PARI problem begs a more complete solution, at least for the docbuild we can get around it by using the build_many from #27490.
This ticket makes some fixes and enhancements to build_many, so that it can also return a result from the function it runs. For the docbuild this feature is not strictly needed, except for the fact that it can also be used (as in Pool.map) to raise any exception that occurs in one of the worker processes. Thus, it's closer in functionality, at least for the purposes of the docbuild, to Pool.map.
CC: @antonio-rojas @jdemeyer @kiwifb @dimpase @saraedum
Component: build
Keywords: docbuild parallel pari
Author: Erik Bray
Branch/Commit: e8d26b6
Reviewer: Dima Pasechnik
Issue created by migration from https://trac.sagemath.org/ticket/28356
Branch does not seem to merge.
Well it used to until a day ago...
Changed branch from u/embray/docbuild/build-many-2 to public/docbuild/build-many-2
Anyone want to go ahead and try using this?
Built docs on Arch against system packages including threaded pari, can confirm it no longer segfaults.
looks good to me.
Reviewer: Dima Pasechnik
Changed branch from public/docbuild/build-many-2 to e8d26b6