covariance matrix is somehow not positive definite
Closed this issue · 4 comments
@davidcannatella has a dataset (reported over in #20: see link to the data here) for which the covariance matrix is not positive defnite. The problem arises at the step of setting the diagonal to 1/4; in this example, the diagonal values are all between 0.2512437 and 0.2512690, but setting the diagonals to 0.25 drops the smallest eigenvalue from 9.788625e-04 to -0.0002842281 (trouble!).
I'll need to remind myself of where the math here is coming from to figure out what to do about it.
So, this is confusing, because we are trying to compute the covariance matrix of a well-defined thing, which should always be positive definite. Ah-ha - the problem is that in the math that led to this, we were using the "population" covariance, i.e., the theoretical covariance of X and Y, where X and Y are the entries in random rows of the frequency matrix. But, cov( ) computes the sample covariance, for which
The denominator n - 1 is used which gives an unbiased estimator of
the (co)variance for i.i.d. observations.
So, a solution is just to multiply cov( ) by (1 - 1/nrow(freqs)). We may still get non-positive-definite matrices in rare cases in the presence of missing data, but we didn't have any guarantees there anyhow (IIRC).
I understand the issue, in the sense that I know the difference between population and sample covariance. Of course I would not have been able to figure this out myself! thanks
When is an updated version of construct likely to be available? If I can do anything to help please let me know.
Dave
When is an updated version of construct likely to be available? If I can do anything to help please let me know.
Thanks! I'm not worried about this as the change is so minor, but I don't want to merge until Gideon has a look at things. You should be able to install the fix by doing
library(devtools)
install_github("gbradburd/conStruct", ref="covariance_fix")
(and it would be helpful if you could confirm that this fixes the problem for you!).