VICReg Loss De-Means Twice?

[RylanSchaeffer]

lightly/lightly/loss/vicreg_loss.py

Lines 128 to 129 in 66ad1b4

	x = x - x.mean(dim=0)
	std = torch.sqrt(x.var(dim=0) + eps)

I think the VICReg loss removes the mean, then calls .var() which also de-means (see: https://pytorch.org/docs/stable/generated/torch.var.html).

If I understand correctly, that seems unnecessary?

[philippmwirth]

Hi @RylanSchaeffer, thank you for bringing this up. I think you are right, the de-mean is only necessary for the covariance term.