The Multiview function does not select the best views for the average prediction
Closed this issue · 2 comments
There following issue is shown here: The Multiview function does not select the k best views for the average prediction
The issue is shown in the example code below:
library(rEDM)
data(block_3sp)
L.3views = Multiview(dataFrame = block_3sp, lib = "1 99", pred = "105 190", E=3,
D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 3)
L.10views = Multiview(dataFrame = block_3sp, lib = "1 99", pred = "105 190", E=3,
D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 10)
L.3views$View[,1:7]
col_1 col_2 col_3 name_1 name_2 name_3 rho
1 1 2 7 x_t(t-0) x_t(t-1) z_t(t-0) 0.9208
2 1 2 6 x_t(t-0) x_t(t-1) y_t(t-2) 0.8677
3 1 2 3 x_t(t-0) x_t(t-1) x_t(t-2) 0.9319
L.10views$View[,1:7]
col_1 col_2 col_3 name_1 name_2 name_3 rho
1 1 2 7 x_t(t-0) x_t(t-1) z_t(t-0) 0.9208
2 1 2 6 x_t(t-0) x_t(t-1) y_t(t-2) 0.8677
3 1 2 3 x_t(t-0) x_t(t-1) x_t(t-2) 0.9319
4 1 2 8 x_t(t-0) x_t(t-1) z_t(t-1) 0.9183
5 1 7 9 x_t(t-0) z_t(t-0) z_t(t-2) 0.8858
6 1 4 9 x_t(t-0) y_t(t-0) z_t(t-2) 0.7774
7 1 3 7 x_t(t-0) x_t(t-2) z_t(t-0) 0.8724
8 1 2 5 x_t(t-0) x_t(t-1) y_t(t-1) 0.9017
9 1 2 4 x_t(t-0) x_t(t-1) y_t(t-0) 0.8805
10 1 3 6 x_t(t-0) x_t(t-2) y_t(t-2) 0.8463
If we compare the two View dataframes above, we see that in the model in which only 3 views are used not the best 3 views are selected (based on rho as described here https://science.sciencemag.org/content/353/6302/922.abstract), because in the model with 10 views we see that there would have been better views to select.
For instance, row 4 should have been selected over row 2.
It follows that the multiview function does no actually select the k best views to perform the average forecasting. It is unclear how it selects them.
Thank you and best regards,
Uriah
sessionInfo(package = "rEDM")
R version 4.0.3 (2020-10-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18363)
Matrix products: default
locale:
[1] LC_COLLATE=English_Switzerland.1252 LC_CTYPE=English_Switzerland.1252 LC_MONETARY=English_Switzerland.1252
[4] LC_NUMERIC=C LC_TIME=English_Switzerland.1252
attached base packages:
character(0)
other attached packages:
[1] rEDM_1.7.3
loaded via a namespace (and not attached):
[1] compiler_4.0.3 graphics_4.0.3 htmltools_0.5.0 tools_4.0.3 utils_4.0.3 yaml_2.2.1 grDevices_4.0.3
[8] Rcpp_1.0.5 stats_4.0.3 datasets_4.0.3 rmarkdown_2.6 knitr_1.30 methods_4.0.3 xfun_0.20
[15] digest_0.6.27 rlang_0.4.10 base_4.0.3 evaluate_0.14
Dear Uriah,
Thank you for your use and analysis of rEDM. I have reproduced your examples, and I think I see what's happening. I see it as a disconnect between the documentation and function behavior. Thank you for pointing this out!
I believe the function is properly reporting the top multiview combinations, the disconnect comes from the value of rho that is reported for each view. More precisely, the value of rho that is used to select the top views may not be the same as that used in the final evaluation of the view, which is reported in the View output table.
This is because the default behavior is to select the top views based on in-sample library and prediction forecast. That is, the default sets pred = lib
when evaluating/selecting the top views. I believe this is what is described in the Ye paper. To be pedantic, if the user specifies lib = "1 10" pred = "11 20"
, then the function evaluates the views with lib = "1 10" pred = "1 10"
, in-sample prediction, but the final output rho is forecast using the specified pred = "11 20"
.
You rightly say: why the disconnect? I don't have a good answer. We need to think about this and how to address it. I suspect it is a good thing to have this flexibility, that the documentation/example should be clarified.
In fact, this disconnect (imo) is one reason the trainLib
argument was added to Multiview
. When True
(default), in-sample prediction sets are used in the selection of the top views, as described above. When trainLib = False
, then the multiview evaluations are performed according to the specified lib
and pred
arguments. This allows one to have multiview select views based on out-of-sample predictions, rather than purely in-sample predictions.
To address/affirm this we can use the default trainLib = True
and set lib = pred
, ala:
> Multiview( dataFrame = block_3sp, lib = "1 99", pred = "1 99", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 3, trainLib = TRUE ) $ View
col_1 col_2 col_3 name_1 name_2 name_3 rho MAE RMSE
1 1 2 7 x_t(t-0) x_t(t-1) z_t(t-0) 0.8875 0.3303 0.4302
2 1 2 6 x_t(t-0) x_t(t-1) y_t(t-2) 0.8834 0.3343 0.4385
3 1 2 3 x_t(t-0) x_t(t-1) x_t(t-2) 0.8692 0.3329 0.4635
which yields the same result as trainLib = FALSE if
lib = pred`
> Multiview( dataFrame = block_3sp, lib = "1 99", pred = "1 99", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 3, trainLib = FALSE ) $ View
col_1 col_2 col_3 name_1 name_2 name_3 rho MAE RMSE
1 1 2 7 x_t(t-0) x_t(t-1) z_t(t-0) 0.8875 0.3303 0.4302
2 1 2 6 x_t(t-0) x_t(t-1) y_t(t-2) 0.8834 0.3343 0.4385
3 1 2 3 x_t(t-0) x_t(t-1) x_t(t-2) 0.8692 0.3329 0.4635
Both of these results select the same top views as the original example, which used default trainLib = TRUE, thereby evaluated the views on lib = pred = 1:99
, but reported final rho using different lib = "1 99"
and pred = "105 190"
:
> Multiview( dataFrame = block_3sp, lib = "1 99", pred = "105 190", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 3, trainLib = TRUE ) $ View
col_1 col_2 col_3 name_1 name_2 name_3 rho MAE RMSE
1 1 2 7 x_t(t-0) x_t(t-1) z_t(t-0) 0.9208 0.2485 0.3164
2 1 2 6 x_t(t-0) x_t(t-1) y_t(t-2) 0.8677 0.3294 0.4113
3 1 2 3 x_t(t-0) x_t(t-1) x_t(t-2) 0.9319 0.2277 0.2934
As long as lib == pred
, the reported values of rho in the output table will reflect the values used in view selection:
Multiview( dataFrame = block_3sp, lib = "20 100", pred = "20 100", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 3, trainLib = TRUE ) $ View
col_1 col_2 col_3 name_1 name_2 name_3 rho MAE RMSE
1 1 2 3 x_t(t-0) x_t(t-1) x_t(t-2) 0.8820 0.3104 0.4226
2 1 2 7 x_t(t-0) x_t(t-1) z_t(t-0) 0.8749 0.3309 0.4306
3 1 3 7 x_t(t-0) x_t(t-2) z_t(t-0) 0.8682 0.3363 0.4411
>
>
> Multiview( dataFrame = block_3sp, lib = "20 100", pred = "20 100", E = 3, D = 3, columns = "x_t y_t z_t", target = "x_t", multiview = 7, trainLib = TRUE ) $ View
col_1 col_2 col_3 name_1 name_2 name_3 rho MAE RMSE
1 1 2 3 x_t(t-0) x_t(t-1) x_t(t-2) 0.8820 0.3104 0.4226
2 1 2 7 x_t(t-0) x_t(t-1) z_t(t-0) 0.8749 0.3309 0.4306
3 1 3 7 x_t(t-0) x_t(t-2) z_t(t-0) 0.8682 0.3363 0.4411
4 1 2 9 x_t(t-0) x_t(t-1) z_t(t-2) 0.8525 0.3514 0.4645
5 1 2 8 x_t(t-0) x_t(t-1) z_t(t-1) 0.8518 0.3362 0.4657
6 1 2 4 x_t(t-0) x_t(t-1) y_t(t-0) 0.8394 0.3485 0.4896
7 1 4 7 x_t(t-0) y_t(t-0) z_t(t-0) 0.8365 0.3749 0.4900
>
I agree that this situation is not stated in the documents.
Thank you for your detailed answer!
The matter is now pretty clear. I think the disconnect makes the function a bit less transparent, so it might be a good idea to store the in-sample rho-values of the views somewhere accessible or to mention this behaviour in the documentation. But it is by no means anything urgent.
Thanks!
Uriah