bwlewis/doRedis

Cannot work with Tune (Tidymodels)

Closed this issue · 5 comments

I tried to use tune_bayes() with doRedis, but it looked like all the workers cannot get any task to work on, on the other hand, the master R process stayed busy and the job queue ("jobs" in my case) remained nil when I checked with redis-cli.

https://www.tidymodels.org/learn/work/bayes-opt/

It seems foreach::getDoParWorkers() cannot return the correct number.

> library(doRedis)
Loading required package: foreach
Loading required package: iterators
> registerDoRedis('debug_jobs')
> foreach::getDoParWorkers()
[1] 0
> startLocalWorkers(n = 2, queue = 'debug_jobs', linger = 5)
> Loading required package: doRedis
Loading required package: doRedis
Loading required package: foreach
Loading required package: foreach
Loading required package: iterators
Loading required package: iterators

> foreach::getDoParWorkers()
[1] 0
> foreach::getDoParWorkers()
[1] 0
> foreach::getDoParWorkers()
[1] 0
> foreach::getDoParWorkers()
[1] 0
> foreach::getDoParWorkers()
[1] 0
> foreach::getDoParWorkers()
[1] 0
> foreach::getDoParWorkers()
[1] 0
> foreach::getDoParWorkers()
[1] 0
> foreach::getDoParWorkers()
[1] 0
> foreach::getDoParWorkers()
[1] 0

Should redisWorker() update .foreachGlobals?

> getDoParRegistered()
[1] TRUE
> getDoParVersion()
[1] "2.0.0"
> getDoParWorkers()
[1] 0
> .foreachGlobals
Error: object '.foreachGlobals' not found
> startLocalWorkers(3, "debug_jobs", linger = 5)
> Loading required package: doRedis
Loading required package: foreach
Loading required package: doRedis
Loading required package: doRedis
Loading required package: foreach
Loading required package: foreach
Loading required package: iterators
Loading required package: iterators
Loading required package: iterators

> getDoParWorkers()
[1] 0
> getDoParWorkers()
[1] 0
> .foreachGlobals
Error: object '.foreachGlobals' not found
> getDoParWorkers
function () 
{
    wc <- if (exists("info", where = .foreachGlobals, inherits = FALSE)) 
        .foreachGlobals$info(.foreachGlobals$data, "workers")
    else NULL
    if (is.null(wc)) 
        1L
    else wc
}
<bytecode: 0x7f8b00218480>
<environment: namespace:foreach>
> .foreachGlobals
Error: object '.foreachGlobals' not found

It seems redisGet() cannot get the right object.

> doRedis:::redisIncr("debug_jobs.count")
[1] 1
> redisGet("debug_jobs.count")
NULL

So this is the root cause. redisGet would return NULL because "3" which is returned by hiredis()$GET() is not raw. Is that a design decision?

uncerealize <- function(x)

Thanks for the fix