gtsherman/cross-validation

Add tie-breaking procedure

Opened this issue · 0 comments

It is possible for more than one parameter setting to tie during the training phase. A tie during the training phase does not, however, imply a tie during the test phase.

Currently, the framework seems to be selecting the first parameter setting alphanumerically, which is essentially random. Instead, to resolve a tie we should:

  • Remove one query (randomly) until the tie is broken
  • If the tie isn't broken, randomly select from the list if ties