Random team selection at Probablistic.multileave
Closed this issue · 2 comments
I wonder whether the current implementation of the random team selection at Probablistic.multileave is correct.
The current code randomizes the order of the team list, use all of them in order, and randomizes it again if all the teams are used.
In "Probabilistic Multileave for Online Retrieval Evaluation, SIGIR 2015", on the other hand, Line 6 in Algorithm 1 may suggest that it draws one team from the team list every time.
How do you think?
My understanding is:
The algorithm first copies all teams R to R' (Line 4).
Each time it randomly draws one team from R' (Line 6), it removes the team from R' (Line 7).
It should be equivalent to shuffling all teams and using each of them one-by-one.
If all teams are used (i.e., R' is empty, Line 5), it copies R to R' again (Line 4).
Sorry, that's my mistake ... The current implementation is correct.