Question about softmax loss

Lines 35 to 46 in 58d06e9

    
           class NCESoftmaxLoss(nn.Module): 
        
               """Softmax cross-entropy loss (a.k.a., info-NCE loss in CPC paper)""" 
        
               def __init__(self): 
        
                   super(NCESoftmaxLoss, self).__init__() 
        
                   self.criterion = nn.CrossEntropyLoss() 
        
               def forward(self, x): 
        
                   bsz = x.shape[0] 
        
                   x = x.squeeze() 
        
                   label = torch.zeros([bsz]).cuda().long() 
        
                   loss = self.criterion(x, label) 
        
                   return loss

Hi, I have a question about using softmax instead of NCE loss.
In that function, every label is set zero including the critic value of positive sample, which has index 0 of the batch.
I want to know the reason. My take on this is that the label should be [1, 0, 0, 0, ...]. Isn't it?

I'm an idiot

@hyBlue seems you found out the answer yourself. Could you explain what NCESoftmaxLoss is trying to do here? I am having the same doubt as you initially had (and feeling like an idiot).

Explanation: label = torch.zeros([bsz]).cuda().long() means the labels are [1, 0, 0, 0, ...] for each sample. label specifies the label of the positive sample, which is 0 in each case.

It's funny how easy it is to get confused even with something you think you are fairly familiar with.

@vinsis Great! You're right

Thanks a lot! (A new idiot)

triple idiot.

	class NCESoftmaxLoss(nn.Module):
	"""Softmax cross-entropy loss (a.k.a., info-NCE loss in CPC paper)"""
	def __init__(self):
	super(NCESoftmaxLoss, self).__init__()
	self.criterion = nn.CrossEntropyLoss()

	def forward(self, x):
	bsz = x.shape[0]
	x = x.squeeze()
	label = torch.zeros([bsz]).cuda().long()
	loss = self.criterion(x, label)
	return loss