raminnakhli/Decoupled-Contrastive-Learning

I doubt the total loss of InfoNCE and DCL should be averaged?

newzhx opened this issue · 0 comments

I think the total loss of InfoNCE as well as DCL should be averaged.
$L=\sum_{k \in { 1,2}, i \in [1,N]}L_{i}^{(k)}$
should be
$L=\frac{1}{N}\sum_{k \in { 1,2}, i \in [1,N]}L_{i}^{(k)}$

and
$L_{DC}=\sum_{k \in { 1,2}, i \in [1,N]}L_{DC,i}^{(k)}$
should be
$L_{DC}=\frac{1}{N}\sum_{k \in { 1,2}, i \in [1,N]}L_{DC,i}^{(k)}$

the code does this

return (positive_loss + negative_loss).mean()