inf Loss

Hi, Thanks for your excellent work!
When I trained on my own dataset, I met the inf loss problem. In several steps starting from 2000 steps, instance_loss_4 and total_loss are inf. After training for a few steps, Loss returned to normal. Is this a normal phenomenon?
Looking forward to your reply, Thank you!

Oh, this is interesting! I've never actually monitored the loss carefully, so I'm not sure if this is normal...

My hunch is that the affinity features are initialized to similar values, so nansum() gets an unexpected high value at the beginning. Then, as the pulling forces re-organize the features (instance_loss_1, instance_loss_2), the pushing loss stabilizes.

garfield/garfield/garfield_model.py

Lines 236 to 240 in d422542

    
           instance_loss_4 = ( 
        
               F.relu( 
        
                   margin - torch.norm(instance[mask[0]] - instance[mask[1]], p=2, dim=-1) 
        
               ) 
        
           ).nansum()

	instance_loss_4 = (
	F.relu(
	margin - torch.norm(instance[mask[0]] - instance[mask[1]], p=2, dim=-1)
	)
	).nansum()