parallell calculation of optimize.portfolio.rebalancing with custom moment functions

[GreenGrassBlueOcean]

If custom moment functions like e.g. CRRA and crra.moments (see also https://cran.r-project.org/web/packages/PortfolioAnalytics/vignettes/custom_moments_objectives.pdf) are used, parallel calculation of optimize.portfolio.rebalancing will fail with the following message: "Optimizer was unable to find a solution for target". This means however not that the optimisation has failed but because the custom moments functions are not known to the parallel (foreach) environment the optimization is not started at all. I have added the attached CRRA.txt (rename to CRRA.R) to the portfolioAnalytics package and recompiled the package from source. After this parallel execution works flawlessly.
This is however only a workaround. I have tried to acomplish a full solution by adding .export = c("CRRA", "crra.moments") argument to the package but this did not work unfortunately.

Furthermore I would recommend if possible to abandon the cryptic message "Optimizer was unable to find a solution for target" by changing the following line in optimize.portfolio.r

minw = try(DEoptim::DEoptim( constrained_objective_v1 ,  lower = lower[1:N] , upper = upper[1:N] ,  control = controlDE, R=R, constraints=constraints, nargs = dotargs , ...=...)) # add ,silent=TRUE here?

   if(inherits(minw,"try-error")) { minw=NULL }
   if(is.null(minw)){
       message(paste("Optimizer was unable to find a solution for target"))
       return (paste("Optimizer was unable to find a solution for target"))
   }

into:

    minw = try(DEoptim::DEoptim( constrained_objective_v1 ,  lower = lower[1:N] , upper = upper[1:N] , control = controlDE, R=R, constraints=constraints, nargs = dotargs , ...=...)) # add ,silent=TRUE here?
 
    if(inherits(minw,"try-error")) { ErrorM <- minw
                                     minw=NULL }
    if(is.null(minw)){
        message(paste("Optimizer was unable to find a solution for target"))
        return (paste("Optimizer was unable to find a solution for target. Error given is:", ErrorM))
    }

It could be considered to also change this by the other optimisation methods
pso::psoptim and GenSA::GenSA as this will make debugging considerably easier

[Caliani21]

@GreenGrassBlueOcean I'm running into the exact same problem. But I couldn't apply your workaround -- I've tried to 1) uninstall the package and remove it from my personal directory; 2) manually recreate from source a new directory destined to the package, along with my custom function text attachment; and 3) change the work directory through setwd() and "recompile" (compile, actually) the package through document(), but it still didn't work. Can you tell me exactly how you managed to pull out that "trick"?

Other than that, when I run chart.RiskReward in an optimize.portfolio.parallel object with optimize_method = "DEoptim" , I obtain the following error message:

Error in subsetx[, risk.column]: incorrect number of dimensions

And I can't seem to plot neither the random portfolios nor the neighbors of the optimal portfolio, which is why I also avoid to run optimize.portfolio in parallel.

@joshuaulrich @braverock Is there any package update coming soon, intended to fix these minor issues? Thanks in advance.

[braverock]

The CRRA vignette builds fine for me, so you need to be more specific about what is failing.

I like the general idea of returning the error that caused the optimizer to fail, though your construction is awkward. I'll think about how to generalize the idea.

[Caliani21]

@braverock But the CRRA vignette doesn't provide an example of optimize.portfolio.rebalancing used in conjunction with a custom function (momentFUN = "sigma.robust", for example) in parallel mode, which is our issue.

My code:

# Registering parallel backend with more workers 
registerDoParallel(detectCores())

# Setting seed
set.seed(8)

# Running the backtest optimization 
bt_mc_opt <- optimize.portfolio.rebalancing(R = cl_returns, portfolio = mc_port_spec, optimize_method = "random", 
                                            trace = TRUE, rp = rp_medium, momentFUN = "custom_moments",
                                            training_period = 261, rolling_window = 261, rebalance_on = "quarters")
            
# Charting the backtest optimization weights
chart.Weights(bt_mc_opt, colorset = cm.colors(ncol(cl_returns)), legend.loc = NULL)

Where cl_returns (cleaned returns) is a xts object obtained with clean.boudt(); mc_port_spec is a portfolio object with mean/CVaR objectives and "full-investment"/long-only/cardinality constraints, rp_medium is a random_portfolios object with sample method; and custom_moments is my custom function for the four moments of the distribution of cl_returns.

This gives me the following error message:

ERROR while rich displaying an object: Error in UseMethod("extractObjectiveMeasures"): 
no applicable method for 'extractObjectiveMeasures' applied to an object of class "c('simpleError', 'error', 'condition')"

Traceback:
1. FUN(X[[i]], ...)
2. tryCatch(withCallingHandlers({
 .     if (!mime %in% names(repr::mime2repr)) 
 .         stop("No repr_* for mimetype ", mime, " in repr::mime2repr")
 .     rpr <- repr::mime2repr[[mime]](obj)
 .     if (is.null(rpr)) 
 .         return(NULL)
 .     prepare_content(is.raw(rpr), rpr)
 . }, error = error_handler), error = outer_handler)
3. tryCatchList(expr, classes, parentenv, handlers)
4. tryCatchOne(expr, names, parentenv, handlers[[1L]])
5. doTryCatch(return(expr), name, parentenv, handler)
6. withCallingHandlers({
 .     if (!mime %in% names(repr::mime2repr)) 
 .         stop("No repr_* for mimetype ", mime, " in repr::mime2repr")
 .     rpr <- repr::mime2repr[[mime]](obj)
 .     if (is.null(rpr)) 
 .         return(NULL)
 .     prepare_content(is.raw(rpr), rpr)
 . }, error = error_handler)
7. repr::mime2repr[[mime]](obj)
8. repr_text.default(obj)
9. paste(capture.output(print(obj)), collapse = "\n")
10. capture.output(print(obj))
11. evalVis(expr)
12. withVisible(eval(expr, pf))
13. eval(expr, pf)
14. eval(expr, pf)
15. print(obj)
16. print.optimize.portfolio.rebalancing(obj)
17. summary(x)
18. summary.optimize.portfolio.rebalancing(x)
19. extractObjectiveMeasures(object)
20. extractObjectiveMeasures.optimize.portfolio.rebalancing(object)
21. unlist(extractObjectiveMeasures(rebal_object[[1]]))
22. extractObjectiveMeasures(rebal_object[[1]])

And when I try to simply run print(bt_mc_opt):

Error in UseMethod("extractObjectiveMeasures"): 
no applicable method for 'extractObjectiveMeasures' applied to an object of class "c('simpleError', 'error', 'condition')"

Traceback:

1. print(bt_mc_opt)
2. print.optimize.portfolio.rebalancing(bt_mc_opt)
3. summary(x)
4. summary.optimize.portfolio.rebalancing(x)
5. extractObjectiveMeasures(object)
6. extractObjectiveMeasures.optimize.portfolio.rebalancing(object)
7. unlist(extractObjectiveMeasures(rebal_object[[1]]))
8. extractObjectiveMeasures(rebal_object[[1]])

Do you have any idea how to get this problem fixed?

[GreenGrassBlueOcean]

Dear both,

The issue is that during the parallel processing the custom objective function is not available in the parallel r session.
That is the reason why the vignette builds (this is single core)

There are two solutions to this issue:

1. Recompile the package including the (exported) custom objective function as a function in the package
2. force single core execution of the rebalancing portfolio. (which can be a terrible experience by large real life portfolios)

I have made a fork of the package in which I choose to add the CRRA function to the package to allow parallel solving for rebalancing CRRA portfolio's.

furthermore I did the following:

• checked all examples and tests
• added new ROI.SYMPHONY plugin (https://github.com/GreenGrassBlueOcean/ROI.plugin.symphony) using a configuration less symphony ROI plugin
• furthermore I made the calculation of randomn portfolio's parallel.
• made a github workflow to test if all test keeps working

You can check it here:
https://github.com/GreenGrassBlueOcean/PortfolioAnalytics

[Caliani21]

I have recently found out that the function clusterExport(), of the package parallel, may be used to export functions of the master R process to each node of the parallel backend (which is much simpler than the recompiling trick):

# Registering parallel backend with snow functionality
cl <- makeCluster(detectCores())
registerDoParallel(cl)

# Exporting custom moments function to registered workers
clusterExport(cl, "custom_moments")