simon-smart88/disagapp

Use ExtendedTask for slow operations

Closed this issue · 2 comments

@timcdlucas I've looked into using ExtendedTask for running slow tasks asynchronously on the async branch. Here's a braindump which we can discuss at some point:

  • It's relatively simple to implement see e.g.: 714067a#diff-6a52e00913fe8cce1f523bc91bdbbcdbe28634dacec3aad6fd59b582c09d392a
  • The logger can be used to keep tabs on when tasks are started and finished which is good, because it's a bit underutilised at the moment.
  • From a UX perspective, there isn't much point in maintaining reactivity for all steps of the analysis, e.g. when fitting the model because there isn't anything else to do at that stage other than wait. There may still be advantages from a performance point of view though with multiple users. It's really the cov, agg and prep_mesh modules where I see this being handy.
  • Calls to the logger can't be used inside the functions any more so these would need converting either to stop for when they are being run outside the app or to return the errors when run in the app so that they can be passed to the logger. For most cov and agg modules, this won't be too much of an issue as there are few things which can go wrong inside the function.
  • SpatRasters need to wrapped to be transferred back, so functions will need an extra parameter (async) adding to handle that when run in the app. For the {disaggregation} functions that are currently used directly, this would mean that they need wrapping in new functions.
  • We'd need to prevent prep modules being used while cov tasks are still running so the tasks need storing as a list in common (haven't checked whether this is possible yet)
  • If used, need to remember to bump up the shiny version in DESCRIPTION.
  • I should probably work on other things first?

Popping this here as an example of the wrapping and unwrapping to make {disaggregation} work with {future}:

covs_prep <- terra::wrap(covs_prep)  
agg <- terra::wrap(agg)
  
prep_remote %<-% {
  prep <- disaggregation::prepare_data(shape, terra::unwrap(covs_prep), terra::unwrap(agg), id_var = "ID_2", response_var = "inc", na.action = TRUE)
  prep$covariate_rasters <- terra::wrap(prep$covariate_rasters)
  prep}

prep_remote_value <- future::value(prep_remote)

prep_remote_value$covariate_rasters <- terra::unwrap(prep_remote_value$covariate_rasters)

For some reason, prep_mesh generates Warning: Error in wk_handle.wk_wkb: Loop 1 edge 9 has duplicate near loop 2 edge 1 when it is run at the same time as other tasks. Works fine with MDG data, so may be a peculiarity of the Scotland data/