The embryonic origin of distinct fat depots and the role for ontogeny in specifying the functional differences among adipocyte lineages between and within depots is unclear. Using a Cre/Lox-based strategy to track the fate of major mesodermal subcompartments in mice we present evidence that <50% of interscapular brown adipocytes are derived from progenitors of the central dermomyotome. Furthermore, we demonstrate that depot-specific adipocyte lineages spatially diverge as early as gastrulation, and that perigonadal adipocytes arise from separate mesodermal subcompartments in males and females. Last, we show adipocyte precursors (APs) of distinct lineages within the same depot exhibit indistinguishable responses to a high fat diet, indicating that ontogenetic differences between APs do not necessarily correspond to functional differences in this context. Altogether, these findings shed light on adipose tissue patterning and suggest that the behavior of adipocyte lineage cells is not strictly determined by developmental history.