Negative pairwise Fst values

[miguel-correa92]

Hi,

I am going through the vignette with a datset of my own and when I plot the matrix of pairwise Fst values I found some negative values (Please see image below). I have 31 accessions and 3 samples of each.

I am just testing the package, so I was wondering what could be the source of this.

Thank you in advance for your time.

Best,

Miguel

[alexviiia]

Negative values arise for close relatives (for example, siblings, first cousins, etc). This is expected from the equations and in practice, so it is not really a bug. In fact, there's a connection between this pairwise FST formula and the KING-robust estimator (for recent/family relatedness) that explains this observation more precisely, but it's not published yet, sorry.

In contrast, zeros are expected for unrelated individuals from the same population (in practice perfect zeroes are impossible, so some of these cases may also be negative due to noise, but magnitudes would be much smaller than what you're finding in your figure), and positive values for individuals of different populations. If the close relationships are not of interest (just the population relationships), feel free to set all negative values to zero; unfortunately there isn't a better way to handle that without stronger assumptions.