"min" method in cal_ranks function: scoring ties in RSN
Closed this issue · 1 comments
Hi, thanks for developing RSN.
I believe this is a ground-breaking model.
I am writing because I saw that the function cal_ranks you use to compute ranks in evaluation function accepts a method parameter that is "min" by default.
If I get it correctly, this conveys the policy for handling ties.
If some entities have the same score as the target one (so if there is a tie) the "min" policy will still give to the target entity the minimum rank. Otherwise, other policies can be applied, e.g. average, or max.
I was wondering how the choice of this policy affects the performances computed for RSN.
Of course, it depends on how many ties RSN generates in evaluation.
In your experience, is this something that happens very often?
Hi, thanks for your attention to our paper.
As you can see in the comment of cal_ranks, the results are usually identical if the model is correctly trained. You can replace the policy with 'average' and find that the result is almost same.
On the other hand, the 'min' policy is implemented by ourselves, and it is much faster compared with others (because we actually do not sort the probs). When we conducted the experiments, we did use the 'average' policy to obtain results only on testing data. But it seems we overlook this in the revision, and we have fixed it.