Setting export compression
benmarchi opened this issue · 4 comments
I was wondering if there was any update on incorporating user defined registry compression in future.batchools (mllg/batchtools#245). It seems like batchtools wants the compression type to be specified in an external configuration file, which is passed as an argument during registry creation. Is it possible for future.batchtools to readily interface with this type of setup or is additional development needed on batchtools?
I've got little time to dive into that, but I'm open for proposals what that could look like and what needs to go on under the hood. It must be designed independently of the Future API itself.
My suggestion would be to allow the user to specify the compression type ("gzip" (default), "xz", FALSE, TRUE (equivalent to "gzip"), etc. as arguments in plan. That way the user can change the default compression behavior using something like:
mybatchtools <- tweak(batchtools_slurm, compress = FALSE)
plan(mybatchtools)
My best guess is that future.batchtools will have to point each future registry to a configuration file with the user specified compression type. Here is a code section from the new test in batchtools linked to registry compression. The first three lines seem to be where the compression specification is happening. If future.batchtools can create and link a configuration file based on the user's plan, it seems like batchtools should be passing the option to all its functions.
An alternative would be to modify the registry object after it has been created by batchtools with the user's compression method. I don't know how exactly this would work because it would need future.batchtools to interact with the future during its evaluation.
Thanks for the layout and for bumping this one. Support for controlling the batchtools 'compress' parameters has been added to the develop branch (commit 9419112)
plan(batchtools_local, registry = list(compress = FALSE))FYI, future.batchtools 0.9.0 which implements this is now on CRAN