Use ExtendedTask for slow operations
Closed this issue · 2 comments
simon-smart88 commented
@timcdlucas I've looked into using ExtendedTask for running slow tasks asynchronously on the async branch. Here's a braindump which we can discuss at some point:
- It's relatively simple to implement see e.g.: 714067a#diff-6a52e00913fe8cce1f523bc91bdbbcdbe28634dacec3aad6fd59b582c09d392a
- The logger can be used to keep tabs on when tasks are started and finished which is good, because it's a bit underutilised at the moment.
- From a UX perspective, there isn't much point in maintaining reactivity for all steps of the analysis, e.g. when fitting the model because there isn't anything else to do at that stage other than wait. There may still be advantages from a performance point of view though with multiple users. It's really the
cov
,agg
andprep_mesh
modules where I see this being handy. - Calls to the logger can't be used inside the functions any more so these would need converting either to
stop
for when they are being run outside the app or to return the errors when run in the app so that they can be passed to the logger. For mostcov
andagg
modules, this won't be too much of an issue as there are few things which can go wrong inside the function. - SpatRasters need to wrapped to be transferred back, so functions will need an extra parameter (
async
) adding to handle that when run in the app. For the{disaggregation}
functions that are currently used directly, this would mean that they need wrapping in new functions. - We'd need to prevent
prep
modules being used whilecov
tasks are still running so the tasks need storing as alist
incommon
(haven't checked whether this is possible yet) - If used, need to remember to bump up the shiny version in DESCRIPTION.
- I should probably work on other things first?
simon-smart88 commented
Popping this here as an example of the wrapping and unwrapping to make {disaggregation}
work with {future}
:
covs_prep <- terra::wrap(covs_prep)
agg <- terra::wrap(agg)
prep_remote %<-% {
prep <- disaggregation::prepare_data(shape, terra::unwrap(covs_prep), terra::unwrap(agg), id_var = "ID_2", response_var = "inc", na.action = TRUE)
prep$covariate_rasters <- terra::wrap(prep$covariate_rasters)
prep}
prep_remote_value <- future::value(prep_remote)
prep_remote_value$covariate_rasters <- terra::unwrap(prep_remote_value$covariate_rasters)
simon-smart88 commented
For some reason, prep_mesh
generates Warning: Error in wk_handle.wk_wkb: Loop 1 edge 9 has duplicate near loop 2 edge 1
when it is run at the same time as other tasks. Works fine with MDG data, so may be a peculiarity of the Scotland data/