mlr-org/mlr3

Feature Request: Allow no required predict sets for a `Measure`

Closed this issue · 1 comments

Some mlr3 Measures don't require a prediction object to operate on, such as msr("selected_features") or msr("internal_valid_score").

Let's say I want to resample xgboost with early stopping and I only care about the internal validation scores (e.g. in the context of tuning). I could then avoid making predictions on the test set alltogether. What I want to do is therefore:

l = lrn("classif.xgboost", nrounds = 100,
  early_stopping_rounds = 10, validate = "test", predict_sets = NULL)
  
rr = resample(tsk("iris"), l, rsmp("holdout")

rr$aggregate(msr("internal_valid_score"))

The reason why the code above does not run is because each measure must have at least one required predict sets because of this check:

self$predict_sets = assert_subset(predict_sets, mlr_reflections$predict_sets, empty.ok = FALSE)

I.e., for MeasureInternalValidScore I need to pick "test", "train" or "internal_valid", even though none are required.
The last call to $aggregate() will therefore err, no matter how I define the MeasureInternalValidScore.

mllg commented

You can toggle empty.ok and check if the unit tests still pass. But to be fair, not wanting to look at predictions at all is an unusual setting ;)