ropensci-books/drake

Need for future.batchtools package

jl5000 opened this issue · 1 comments

I'm creating a drake project using parallel processing (locally, not using a cluster) and the user manual has left me a bit uncertain as to which packages I actually require. The future package is directed to be installed, so that is clear. However, I'm not sure if "be sure to install future.batchtools" is a necessity or a suggestion. If it is required, it's also not clear whether it needs to be loaded with library() or just needs to be installed.

future.batchtools is only required for make(parallelism = "future") on clusters, and no library() call is required for either future or future.batchtools. In your case, you do not need future.batchtools. However, the future.callr package has more reliable backend infrastructure than the default backends.

# install.packages("future")
# install.packages("future.callr")
future::plan(future.callr::callr, workers = 2)
make(plan, parallelism = "future", jobs = 2)

Even better, clustermq has a multicore backend, and for drake I think it will be your fastest option.

# install.packages("clustermq")
options(clustermq.scheduler = "multicore")
make(plan, parallelism = "clustermq", jobs = 2)