Deterministic or random order at equal rating.
Closed this issue · 3 comments
I received this request by email
Although your wiki provides very much information on the use itself, it does not clarify how the question selection strategy (number of questions to draw) works in detail. We've been having difficulty figuring that out. For example, let's say we have the following setup:
Number of questions in list: 200 (all rated 300)
User rating: 350
User win probability: 0.75
Number of questions to draw: 5
What does the algorithm do now? As all questions are rated the same, it should (from my understanding) select 5 random questions out of the whole list of 200. However, this does not seem to happen. Instead, it seems as if the user always gets the first 5 questions in the same order they appear in the question list.
And what does the algorithm do then after the first question was answered? What does it do after 5 questions were answered? The problem that we have experienced here is that the user seems to be getting the same handful of questions over and over again. In order to avoid that same question problem, we have also tried it with '200 questions to draw'. In that case, it should definitely select the questions randomly out of the whole list of 200, but this does not happen either :-).
I hope I could make myself clear so far and would be happy if you could give me some more details about how the underlying algorithm works.
If you need me to do some further testing, just let me know and I'll provide you with more detailed numbers/tests.
Keep up the good work and have a good weekend.
The problem seems to the random selection from the shortlist, since behaviour appears deterministic also when the shortlist is 200. We need to check this in the code.
The shortlist should be selected deterministically. Randomising is expensive, and the case of equal ratings is supposed to be rare.
The first behaviour, that the same five questions are drawn every time, is expected when all the questions have the same rating. The design has assumed that this will not be the case, and simply take the N first questions ordered by rating difference. If questions tie for a place among the N, they come in DB order,. Randomising at this stage would be expensive.
I do not yet understand why the same thing happens when the shortlist is 200 questions.
We are unable to reproduce the bug. Closing.
There is a use case for randomising tthe selection from tied questions, but this should be a separate issue.