GiancarloValente/MVPACrossValidationAndPermutations

Question about permutation test run times with repeat cross validation

Opened this issue · 0 comments

Hi!

I have read your great paper Cross-validation and permutations in MVPA: Validity of permutation strategies and power of cross-validation schemes and learned a lot, but I still have a question about permutation run times with repeat cross validation .

The question is: If my repeated cross-validation uses more repetitions, will I need more runs during permutation test?

For example, if I have finished a 10-fold cross-validation with 20 repeats (in this time, I have trained model 10 fold * 20 repeats = 200 times), and then, I want to run permutation test 1000 times, should I train model 1000 * 10 * 20 = 200000 times, or 1000 * 10 times for permutation test?

Looking forward to your reply, thanks!