Cov matrix used instead of precision matrix?

[Parskatt]

At:

flowgmm/flow_ssl/distributions.py

Lines 22 to 25 in 422ff5d

	def gaussians(self):
	gaussians = [distributions.MultivariateNormal(mean, F.softplus(inv_std)**2 * torch.eye(self.d).to(self.device))
	for mean, inv_std in zip(self.means, self.inv_cov_stds)]
	return gaussians

You seem to insert a precision matrix, but reading the documentation:

https://github.com/pytorch/pytorch/blob/8b248af35d43c97d0e437f6f4ff0fbd4da5700c8/torch/distributions/multivariate_normal.py#L119

It seems like if you do not specify the matrix as a precision matrix, it will be assumed to be a covariance matrix (by argument order).

Is this intended?

[izmailovpavel]

Hi @Parskatt, thank you for noticing this! I believe you are correct, and this was not intended. Practically, I believe this does not make a difference, but we need to rename the parameter.

[Parskatt]

Yeah, probably it doesn't matter since you initialize inv_std so that the softplus puts it at 1. Maybe its slightly easier to get a singular distribution (i.e. close to zero variance) with the covariance parameterization, don't think it should be too bad though :)

[izmailovpavel]

Right :) We are also typically not training these covariance parameters and just keep them fixed at 1. throughout. Definitely a mistake on our side though.