Add easy function for grouped df processing

[orgadish]

Performing grouped df processing is an excellent opportunity for parallelization, but can be cumbersome to think about and do for a basic user. However, the groupings allow an opportunity to automate the splitting by workers and thus abstract the parallelization from the users.

Here is a (non-optimized) function I wrote that makes it easy to take normal df processing calls and parallelize them by number of workers.

parallelized_df_do <- function(.data, .f, ..., n_workers=NULL) {
  f(is.null(n_workers)) n_workers <- future::nbrOfWorkers()
  
  keys_tbl <- .data %>% 
    dplyr::group_keys() %>% 
    dplyr::mutate(.worker_id = sample(1:n_workers, replace=T, size=nrow(.)))
  
  .data %>% 
    dplyr::ungroup %>% 
    dplyr::left_join(keys_tbl, by = dplyr::group_vars(.data)) %>% 
    dplyr::group_split(.worker_id, .keep=F) %>% 
    map(~dplyr::group_by(.x, !!!dplyr::groups(.data))) %>% 
    furrr::future_map_dfr(.f, ...)
}

# An example
iris %>% 
  group_by(Petal.Length, Species) %>% 
  parallelized_df_do(~summarize(.x, across(starts_with("Sepal"), sum)))

[DavisVaughan]

furrr is extremely focused on only exposing parallel versions of the purrr API, so I don't think this is a good fit for this package.

Have you seen multidplyr? It attempts to do some of this. https://github.com/tidyverse/multidplyr

[orgadish]

Thanks @DavisVaughan.
I hadn't heard about multidplyr, and I wasn't sure where this particular utility function should go. I agree that it's outside the traditional scope of furrr. And I see some of it (if not all) has already been implemented in multidplyr.

Thanks for pointing me in that direction!