Add tie-breaking procedure
Opened this issue · 0 comments
gtsherman commented
It is possible for more than one parameter setting to tie during the training phase. A tie during the training phase does not, however, imply a tie during the test phase.
Currently, the framework seems to be selecting the first parameter setting alphanumerically, which is essentially random. Instead, to resolve a tie we should:
- Remove one query (randomly) until the tie is broken
- If the tie isn't broken, randomly select from the list if ties