An allele-sharing, moment-based estimator of global, population-specific and population-pair F_ST under a general model of population structure

by Jerome Goudet, Bruce S. Weir

Being able to properly quantify genetic differentiation is key to understanding the evolutionary potential of a species. One central parameter in this context is F_ST, the mean coancestry within populations relative to the mean coancestry between populations. Researchers have been estimating F_ST globally or between pairs of populations for a long time. More recently, it has been proposed to estimate population-specific F_ST values, and population-pair mean relative coancestry. Here, we review the several definitions and estimation methods of F_ST, and stress that they provide values relative to a reference population. We show the good statistical properties of an allele-sharing, method of moments based estimator of F_ST (global, population-specific and population-pair) under a very general model of population structure. We point to the limitation of existing likelihood and Bayesian estimators when the populations are not independent. Last, we show that recent attempts to estimate absolute, rather than relative, mean coancestry fail to do so.