Question about permutation test run times with repeat cross validation
Opened this issue · 0 comments
Bigsealion commented
Hi!
I have read your great paper Cross-validation and permutations in MVPA: Validity of permutation strategies and power of cross-validation schemes and learned a lot, but I still have a question about permutation run times with repeat cross validation .
The question is: If my repeated cross-validation uses more repetitions, will I need more runs during permutation test?
For example, if I have finished a 10-fold cross-validation with 20 repeats (in this time, I have trained model 10 fold * 20 repeats = 200 times), and then, I want to run permutation test 1000 times, should I train model 1000 * 10 * 20 = 200000 times, or 1000 * 10 times for permutation test?
Looking forward to your reply, thanks!