Training-data mixtures are product priorities expressed as weights

I think training-data mixtures are product priorities expressed as weights. A percentage assigned to code, books, multilingual web text, mathematics, support conversations, or synthetic instructions is not a preprocessing detail. It is a decision about which users, tasks, and failure modes receive a share of finite model capacity. Teams may discuss the mixture as if it were a neutral recipe discovered by the training group. In practice, it is the product strategy made numerical—and it deserves the same explicit ownership as a roadmap.

The raw corpus does not settle the question. Public datasets embody different philosophies: broad domain diversity, aggressively filtered educational material, annotation-rich multilingual coverage, or competition-driven quality selection. Sampling them in proportion to their available token counts simply lets collection economics choose the product. Web text dominates because it is abundant, not because the product should reproduce the web’s distribution. Oversampling code may improve structured reasoning beyond programming; emphasizing books may deepen long-form coherence; raising a language’s share may serve a market while reducing capacity elsewhere. Every gain has an opportunity cost.

That is why I would start mixture design with a capability portfolio. Name the behaviours the product needs, the populations it must serve, the risks it must contain, and the evaluations that represent each one. Then map data domains to those outcomes, admitting uncertainty instead of treating source labels as causal truth. A “math” bucket may include worked reasoning, copied answers, and noisy notation. A “high-quality” classifier trained on Wikipedia can quietly encode stylistic narrowness. Synthetic data can target scarce capabilities and formats, but one generator can concentrate the distribution and create a closed lineage loop. The labels are hypotheses that require ablation.

Methods such as DoReMi and RegMix improve the search for domain weights, but they do not remove product judgment. A minimax objective protects weak domains according to the groups someone chose. A regression objective optimizes the benchmarks someone selected. A cooldown phase that spends the final compute on educational or instruction data privileges a particular final impression. Optimization can make the allocation evidence-led; it cannot decide what the allocation should value. The objective function is the strategy document.

I want mixtures versioned as durable data products: source manifests, licences, provenance, filter versions, dedup policy, synthetic-generator lineage, domain weights, curriculum schedule, and the evaluation evidence behind each change. A model release should be traceable to that record. When a capability regresses or a right must be revoked, the team should know whether the relevant examples entered through a source, a filter, a weight, or a synthetic branch. Otherwise each retraining run is a fresh act of organizational amnesia.

I concede one narrow case: a small team adapting an existing model for a tightly bounded task may reasonably use a simple, hand-built mixture rather than run hundreds of proxy-model experiments. The boundary is not rigor versus informality. It is whether the team writes down the intended behaviour, checks coverage and contamination, preserves lineage, and evaluates the resulting trade. Small scale reduces the search cost; it does not make weights neutral.

A product roadmap allocates engineering time among promised outcomes. A data mixture allocates learning signal among promised behaviours. I see them as the same management act at different layers. If product leaders cannot explain why one domain receives ten times another, then the most consequential prioritization in the model is happening without accountable ownership.