topepo/caret

Custom summary measure with OOB resampling?

Opened this issue · 0 comments

I am trying to use caret to fit a random forest binary classification model using OOB resampling with a custom summary measure.
However, I get the following warning:

library(caret);

### custom performance metric function
# https://stackoverflow.com/a/52697940
mySummary  <- function (data, lev = NULL, model = NULL) {
    a1 <- defaultSummary(data, lev, model) # accuracy, kappa
    b1 <- twoClassSummary(data, lev, model) # area under ROC, sens, spec
    c1 <- prSummary(data, lev, model) # area under PR curve, prec, recall, F-score
    out <- c(a1, b1, c1);
    return(out);
  }

# training options
seed = 123
train.ctrl.rf <- trainControl(
    method = "oob",
    classProbs = TRUE,
    summaryFunction = mySummary
    );

Warning message:
Custom summary measures cannot be computed for out-of-bag resampling. This value of summaryFunction will be ignored.

Is there any plan to support custom summary measures with OOB resampling?