controllable stage parallelism
conderls opened this issue · 2 comments
Is your feature request related to a problem? Please describe.
the stage parallelism is enabled by new Stage().parallel(true)
and implemented by par collections
private[this] def parallelFactories: Either[ParArray[Factory[_]], Array[Factory[_]]] = {
if (_parallel) {
Left(factories.par)
} else {
Right(factories)
}
}
to avoid Java Heap Space OOM, I think it is a good idea to set parallelism level.
Describe the solution you'd like
maybe it can be achieved with the thread pool executor:
with option new Stage().parallel(2)
to control parallelism level = 2;
https://docs.scala-lang.org/overviews/parallel-collections/configuration.html
Working on it :) It seems that the type signature of new ForkJoinTaskSupport()
has been changed from scala 2.11 to 2.12, which prevents us from compiling with scala 2.11. I will try to bypass this issue by using reflection
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.