TreeMix is a method for inferring the patterns of population splits and mixtures in the history of a set of populations. In the underlying model, the modern-day populations in a species are related to a common ancestor via a graph of ancestral populations. We use the allele frequencies in the modern populations to infer the structure of this graph.
The details of the TreeMix model are presented in: Pickrell JK and Pritchard JK, Inference of population splits and mixtures from genome-wide allele frequency data.