Note: The following work titled A Comparison of Random Forest Variable Selection Methods for Regression Modeling of Continuous Outcomes by O’Connell, N.S., Jaeger, B.C., Bullock, G.S., and Speiser, J.L. is currently submitted for peer review publication.
The goal of rfvs-regression is to compare random forest variable selection techniques for continuous outcomes. We compare several methods available through various R packages and user defined functions from published paper appendices. The code for implementing each RF variable selection approach tested can be found in the function “rfvs()” and each of the methods assessed and compared are given in the following table:
| Abbreviation in Paper | Publication | R package | Approach | Type of forest method |
|---|---|---|---|---|
| None | Breiman 2001 | ranger | N/A | Axis |
| Svetnik | Svetnik 2004 | Uses party, code from Hapfelmeier | Performance Based | Conditional Inference |
| Jiang | Jiang 2004 | Uses party, code from Hapfelmeier | Performance Based | Conditional Inference |
| Caret | Kuhn 2008 | caret | Performance Based | Axis |
| Altmann | Altmann 2010 | vita | Test Based | Axis |
| Boruta | Kursa 2010 | Boruta | Test Based | Axis |
| aorsf - Menze | Menze 2011 | aorsf | Performance Based | Oblique |
| RRF | Deng 2013 | RRF | Performance Based | Axis |
| SRC | Ishwaran 2014 | randomForestSRC | Performance Based | Axis |
| VSURF | Genuer 2015 | VSURF | Performance Based | Axis |
| aorsf-Negation | Jaeger 2023 | aorsf | Performance Based | Oblique |
| aorsf- Permutation | Jaeger 2023 | aorsf | Performance Based | Oblique |
| Axis - SFE | NA | ranger | Test Based | Axis |
| rfvimptest | Hapfelmeier 2023 | rfvimptest | Test Based | Conditional Inference |
In this benchmarking study, we pulled datasets from OpenML and modeldata following the criteria and steps outlined below:
A total of 59 datasets met criteria and were used in this benchmarking study. Summary characteristics of these datasets are given in the figure below
We used five replications of split sample validation (i.e., Monte-Carlo cross validation) for each dataset to evaluate RF variable selection methods.
-
First, a dataset was split into training (75%) and testing (25%) sets.
-
Second, each variable selection method was applied to the training data, and the variables selected by each method were saved.
-
Third, a standard axis-based RF model using the R package ranger and an oblique RF using the package aorsf were fit on the training data set using variables selected from each method, and R^2 was recorded based on the test data for each replication, method, and dataset.
-
Fourth, methods of variable selection were compared based on computation time, accuracy measured by R^2, and percent variable reduction
note: If any missing values were present in the training or testing data, they were imputed prior to running variable selection methods using the mean and mode for continuous and categorical predictors, respectively, computed in the training data.
We provide the results in the table below for R^2 for downstream models fitted in Axis and Oblique RFs, variable percent reduction (higher % reduction implies more variables eliminated on average), and computation time (in seconds).
| R-Squared (Axis) | R-Squared (Oblique) | Variable Percent Reduced | Time (seconds) | |
|---|---|---|---|---|
| Altman | ||||
|
Mean (SD) Median \[IQR\] |
0.41 ( 0.017 ) 0.39 \[ 0.183 , 0.626 \] |
0.43 (0.018) 0.44 \[0.223, 0.651\] |
0.75 ( 0.012 ) 0.82 \[ 0.667 , 0.9 \] |
338.02 ( 37.636 ) 52.63 \[ 19.956 , 379.291 \] |
|
aorsf - Menze |
||||
|
Mean (SD) Median \[IQR\] |
0.42 ( 0.018 ) 0.42 \[ 0.212 , 0.693 \] |
0.45 (0.018) 0.47 \[0.233, 0.724\] |
0.61 ( 0.015 ) 0.67 \[ 0.439 , 0.812 \] |
17.8 ( 2.427 ) 3.57 \[ 1.198 , 12.561 \] |
|
aorsf - Permutation |
||||
|
Mean (SD) Median \[IQR\] |
0.42 ( 0.019 ) 0.4 \[ 0.22 , 0.694 \] |
0.45 (0.019) 0.48 \[0.236, 0.724\] |
0.56 ( 0.016 ) 0.6 \[ 0.364 , 0.8 \] |
42.67 ( 6.464 ) 4.91 \[ 1.145 , 24.119 \] |
| Boruta | ||||
|
Mean (SD) Median \[IQR\] |
0.41 ( 0.019 ) 0.42 \[ 0.208 , 0.677 \] |
0.43 (0.018) 0.44 \[0.225, 0.673\] |
0.46 ( 0.019 ) 0.45 \[ 0.133 , 0.774 \] |
40.58 ( 4.363 ) 8.35 \[ 3.519 , 47.411 \] |
| CARET | ||||
|
Mean (SD) Median \[IQR\] |
0.43 ( 0.019 ) 0.42 \[ 0.23 , 0.694 \] |
0.44 (0.019) 0.44 \[0.218, 0.681\] |
0.48 ( 0.02 ) 0.5 \[ 0.111 , 0.826 \] |
3544.69 ( 580.392 ) 171.95 \[ 52.846 , 1170.799 \] |
| rfvimptest | ||||
|
Mean (SD) Median \[IQR\] |
0.18 ( 0.017 ) 0.08 \[ -0.005 , 0.32 \] |
0.21 (0.017) 0.1 \[0, 0.401\] |
0.92 ( 0.002 ) 0.93 \[ 0.9 , 0.95 \] |
1068.56 ( 151.485 ) 143.83 \[ 35.449 , 563.047 \] |
| Jiang | ||||
|
Mean (SD) Median \[IQR\] |
0.42 ( 0.019 ) 0.42 \[ 0.22 , 0.693 \] |
0.44 (0.019) 0.44 \[0.232, 0.72\] |
0.65 ( 0.015 ) 0.7 \[ 0.474 , 0.872 \] |
455.36 ( 63.503 ) 41.57 \[ 14.373 , 260.241 \] |
| SRC | ||||
|
Mean (SD) Median \[IQR\] |
0.41 ( 0.018 ) 0.39 \[ 0.199 , 0.666 \] |
0.41 (0.018) 0.42 \[0.143, 0.634\] |
0.35 ( 0.019 ) 0.27 \[ 0 , 0.667 \] |
30.08 ( 1.16 ) 20.87 \[ 13.91 , 47.381 \] |
|
aorsf - Negation |
||||
|
Mean (SD) Median \[IQR\] |
0.41 ( 0.019 ) 0.4 \[ 0.182 , 0.671 \] |
0.44 (0.019) 0.41 \[0.188, 0.726\] |
0.53 ( 0.016 ) 0.58 \[ 0.325 , 0.776 \] |
44.32 ( 7.204 ) 5.06 \[ 1.166 , 20.333 \] |
| None | ||||
|
Mean (SD) Median \[IQR\] |
0.4 ( 0.017 ) 0.4 \[ 0.182 , 0.621 \] |
0.4 (0.017) 0.4 \[0.137, 0.635\] |
0 ( 0 ) 0 \[ 0 , 0 \] |
0 ( 0 ) 0 \[ 0 , 0 \] |
|
Axis - SFE |
||||
|
Mean (SD) Median \[IQR\] |
0.4 ( 0.018 ) 0.39 \[ 0.196 , 0.624 \] |
0.4 (0.017) 0.4 \[0.161, 0.633\] |
0.13 ( 0.01 ) 0.05 \[ 0 , 0.243 \] |
0.37 ( 0.032 ) 0.08 \[ 0.041 , 0.373 \] |
| RRF | ||||
|
Mean (SD) Median \[IQR\] |
0.4 ( 0.017 ) 0.4 \[ 0.17 , 0.623 \] |
0.4 (0.017) 0.39 \[0.137, 0.633\] |
0.02 ( 0.004 ) 0 \[ 0 , 0 \] |
2.43 ( 0.256 ) 0.6 \[ 0.152 , 2.735 \] |
| Svetnik | ||||
|
Mean (SD) Median \[IQR\] |
0.41 ( 0.018 ) 0.37 \[ 0.181 , 0.642 \] |
0.43 (0.018) 0.4 \[0.204, 0.68\] |
0.69 ( 0.016 ) 0.76 \[ 0.577 , 0.904 \] |
1197.21 ( 134.788 ) 245.47 \[ 76.233 , 1317.819 \] |
| VSURF | ||||
|
Mean (SD) Median \[IQR\] |
0.43 ( 0.018 ) 0.41 \[ 0.205 , 0.69 \] |
0.43 (0.019) 0.43 \[0.216, 0.714\] |
0.76 ( 0.014 ) 0.84 \[ 0.707 , 0.918 \] |
256.19 ( 41.346 ) 23.71 \[ 8.851 , 125.959 \] |
We present the results of accuracy, by time, and percent reduction in the figure below. K-Means clustering was used to find the cluster of methods that perform best optimally in terms of computation time and accuracy (in the bottom right corner of the figure), with size and color denoting percent reduction.
We observe that for downstream Axis forests fitted in ranger, the methods of Boruta (r package: boruta) and aorsf-Menze (r package: aorsf) perform optimmaly in terms of fast computation time and high-accuracy while preserving good parsimony (good variable percent reduction).
For downstream oblique forests fitted in aorsf, the methods aorsf-Menze and aorsf-Permutation (both found within the aorsf R package) perform best in terms of computation time and accuracy.
We note that in several dataset replications, at least one method failed to select a single variable in variable selection. We performed a sensitivity analysis by assessing results only in replications where all methods selected at least one variable (2,464 out of 4,260)
Last but not least, we compare downstream fitted Axis RFs to Oblique RFs.
We find that in terms of median accuracy, downstream oblique fitted forests generally perform slightly better than downstream Axis forests, particularly among the top performing methods of variable selection.