Should we randomly change the order of rel_doc, irr_doc pair?

[george-pch806771]

Hi,
In my training data, there are two columns rel_doc and irr_doc,
which means the ranking of doc in pos_doc should be higher than the one in irr_doc.
If the model input is a (rel_doc, irr_doc) pair,
then all of the label(ground truth) will be just 1.
Should we randomly change the order of rel_doc, irr_doc pair and convert the label to -1?

ps. In the paper From RankNet to LambdaRank to
LambdaMART: An Overview, it say the cost function is symmetric while swapping i,j and change label(S_ij) from 1 to -1,
but in my case, the S_ij is fixed(S_ij=1 or say S_ji=-1).

[airalcorn2]

You don't need to shuffle the order of the documents because the architecture is a Siamese network, i.e., the same network processes both relevant and irrelevant documents. And the cost function is indeed symmetric because the cost is calculated from the difference in the scores. Here's a little NumPy example:

import numpy as np

# Document i's score.
si = 0.6
# Document j's score.
sj = 0.2
# Score differences.
oij = si - sj
oji = sj - si

# See Equation (3) in "Learning to Rank using Gradient Descent":
# https://icml.cc/2015/wp-content/uploads/2015/06/icml_ranking.pdf.

# Cij when i is more relevant than j.
print(-1 * oij + np.log(1 + np.exp(oij)))
# Same as Cji when i is more relevant than j.
print(-0 * oji + np.log(1 + np.exp(oji)))

# Cij when j is more relevant than i.
print(-0 * oij + np.log(1 + np.exp(oij)))
# Same as Cji when j is more relevant than i.
print(-1 * oji + np.log(1 + np.exp(oji)))

[george-pch806771]

I get it now, thank you for your explanation!