On-demand branch creation
Closed this issue · 11 comments
Dynamic branches take up a lot of memory in the main session of a large pipeline. Instead of a full branch object, it may be possible to store a lightweight reference to the branch until it is actually needed. If this works, we may see a large reduction in memory consumption.
Bud targets probably also need a similar treatment.
Actually, I think it's more efficient to just go with the original serialization idea proposed in #1352. Even after subtracting pedigree creation time, it is much faster to deserialize a branch on demand than to create one from scratch.
command <- command_init()
settings <- settings_init()
cue <- cue_init()
value <- value_init()
branch <- branch_init(command, settings, cue, value)
serialized_branch_high <- qs::qserialize(branch, preset = "high")
serialized_branch_balanced <- qs::qserialize(branch, preset = "balanced")
serialized_branch_fast <- qs::qserialize(branch, preset = "fast")
microbenchmark(
create_branch = branch_init(command, settings, cue, value),
create_pedigree = pedigree_new(parent = branch$settings$name, index = 1L),
deserialize_high = qs::qdeserialize(serialized_branch_high),
deserialize_balanced = qs::qdeserialize(serialized_branch_balanced),
deserialize_fast = qs::qdeserialize(serialized_branch_fast),
times = 1e4,
control = list(warmup = 100)
)
#> Unit: microseconds
#> expr min lq mean median uq max neval cld
#> create_branch 56.621 58.876 68.514920 60.721 64.9235 8143.133 10000 a
#> create_pedigree 1.558 2.009 2.270166 2.173 2.2960 34.440 10000 b
#> deserialize_high 17.917 18.942 27.222868 21.689 34.3170 7516.366 10000 c
#> deserialize_balanced 14.801 15.785 23.145619 18.204 30.8730 403.645 10000 d
#> deserialize_fast 14.760 15.662 22.881981 17.917 30.7090 204.795 10000 dand to summarize the storage sizes in the various options:
command <- command_init()
settings <- settings_init(name = "target_name")
cue <- cue_init()
value <- value_init()
branch <- branch_init(command, settings, cue, value, index = 1L)
serialized_branch_high <- qs::qserialize(branch, preset = "high")
serialized_branch_balanced <- qs::qserialize(branch, preset = "balanced")
serialized_branch_fast <- qs::qserialize(branch, preset = "fast")
library(lobstr)
obj_size(qs::qserialize(branch$pedigree))
#> 176 B
obj_size(branch$pedigree)
#> 456 B
obj_size(serialized_branch_high)
#> 680 B
obj_size(serialized_branch_balanced)
#> 840 B
obj_size(serialized_branch_fast)
#> 1.17 kB
obj_size(branch)
#> 9.54 kBThe "high" present on the branch looks like a good tradeoff (#1365).
After optimizing with #1368, branch creation got much faster. Also, it will be much easier now to create branches on demand and store only lightweight references whenever possible. I will need to refactor the junction class and add fancy checking to pipleine_set_target() and pipeline_get_target(), but #1368 also makes this part easier.
target <- tar_target(y, x, pattern = map(x))
name <- "x_branch"
command <- target$command
store <- target$store
cue <- target$cue
settings <- target$settings
index <- 1L
deps_parent <- character(0L)
deps_child <- character(0L)
branch <- branch_init(
name = name,
command = command,
deps_parent = deps_parent,
deps_child = deps_child,
settings = settings,
cue = cue,
store = store,
index = index
)
serialized_branch_high <- qs::qserialize(branch, preset = "high")
serialized_branch_balanced <- qs::qserialize(branch, preset = "balanced")
serialized_branch_fast <- qs::qserialize(branch, preset = "fast")
microbenchmark::microbenchmark(
create_branch = branch_init(
name = name,
command = command,
deps_parent = deps_parent,
deps_child = deps_child,
settings = settings,
cue = cue,
store = store,
index = index
),
deserialize_high = qs::qdeserialize(serialized_branch_high),
deserialize_balanced = qs::qdeserialize(serialized_branch_balanced),
deserialize_fast = qs::qdeserialize(serialized_branch_fast),
times = 1e4,
control = list(warmup = 100)
)
#> Unit: microseconds
#> expr min lq mean median uq max neval cld
#> create_branch 15.170 16.400 19.10258 16.974 18.204 5561.978 10000 a
#> deserialize_high 17.835 19.024 26.08239 20.623 28.618 6385.299 10000 b
#> deserialize_balanced 14.678 15.785 21.73968 16.851 25.092 6344.299 10000 a
#> deserialize_fast 14.555 15.662 20.98209 16.687 24.928 5613.187 10000 a Notes to self on the next steps for the implementation:
- ensure values when the subpipeline is created. For transient-memory targets, ensure the value after making the subpipeline copy. For persistent memory targets, do it before.
- The lightweight reference in the pipeline object should include the parent name, the file path if known, and the file stage if known.
- Transient memory branches should be converted back to references once they run. Persistent memory branches should not.
- Need to look at if/how buds keep persistent values in memory
- need to look at target_load_value on patterns. Can those targets be converted back into references?
targets already has a sophisticated mechanism for transient memory via pipeline_unload_transient(). I think it's just a matter of converting the target definition object back to a reference during pipeline_unload_target().
In 23652fd (branch 1364), I added a new R/class_refernece.R class and unit tests in tests/testthat/test-class_reference.R. This is the new machinery for converting branches and buds to and from lightweight references. At this point, I just need to plug reference_produce_target() into pipeline_get_target() and target_produce_reference() into pipeline_unload_target(). (pipeline_set_target() should not change.) And for efficiency, I should reduce excessive calls to pipeline_get_target(), which I have treated thus far as inexpensive.
I also need to go into the pattern and stem classes and make sure they store references and not whole targets in the pipeline object when they create branches and buds.
83b706c converts branches and buds to and from lightweight references using the existing machinery of target_load_value() and pipeline_unload_target(). Next, I will ensure branches and buds are created as references from the start.
branch 1364 implements the above, and branch 1364-ref3 takes the extra step of creating branches and buds as lightweight references from the beginning. The latter is failing at the moment and I have to step away for now, but I will return to it soon.