Training scale should be governed by data yield, not accelerator count

I think training scale should be governed by data yield, not accelerator count. A larger cluster tells me how much work a system can attempt; it does not tell me how much useful learning the run extracts from its corpus. The LLM training pipeline makes the underlying asymmetry clear: pretraining consumes almost all training compute, while instruction tuning, reward modelling, and RLHF are comparatively cheap to repeat. That makes a weak pretraining run an unusually expensive way to discover that the data was redundant, poorly mixed, or already exhausted.

Infrastructure measures serve a different purpose. Model FLOPs utilisation exposes pipeline bubbles, communication overhead, memory-bound operations, recomputation, and data-loading stalls. Checkpointing protects weeks of work. Mixed precision, tensor parallelism, pipeline parallelism, and prefetching turn theoretical hardware into actual token throughput. I use those controls to determine whether the training system is efficient. They cannot tell me whether the model is learning something worth buying. A run can achieve strong utilisation while repeatedly presenting low-value tokens.

I would therefore make data yield a release metric for training runs. At minimum, I want the validation-loss improvement attributable to each tranche of tokens, measured against the GPU-hours and money used to process it. For product-facing decisions, I would add capability yield: the change in a stable, versioned evaluation portfolio per billion accepted tokens. Neither number should be collapsed into one universal score. Loss reveals broad predictive learning; capability evaluations reveal whether that learning reaches the behaviours the model is meant to support. Their disagreement is evidence, not noise to average away.

This changes the scale-up gate. Before adding accelerators, I would run smaller probes across candidate sources, filters, mixtures, and curricula. Each probe should preserve a manifest of accepted data, deduplication policy, source weights, tokenizer version, sequence length, model configuration, and evaluation deltas. The question is not “Can the cluster process twice as many tokens?” It is “Which next tokens produce enough marginal improvement to justify processing them at full scale?” Scaling laws say that more data and more parameters can both reduce loss, with diminishing returns. They do not turn arbitrary volume into useful signal.

The same rule prevents a category error around model size. A larger model needs more memory, more distributed coordination, and more training compute; it can also be undertrained if the token supply does not scale with it. Accelerator count can hide that mismatch because the run looks ambitious and the throughput graph looks healthy. Data yield makes the mismatch visible. If fresh tranches stop moving validation loss or the target evaluations, I should question corpus composition, quality filters, and parameter-to-token allocation before reserving another rack.

I would review yield as a curve, not just an end-of-run ratio. Early and late checkpoints can show whether a source contributes quickly and then saturates, whether a curriculum changes the slope, or whether a mixture improves one capability while displacing another. Learning-rate schedules, gradient clipping, validation tracking, and intermediate checkpoints matter here because optimisation instability can masquerade as poor data. A yield review must distinguish data exhaustion from a bad schedule, a damaged update, or an infrastructure stall before changing the corpus.

I concede one narrow boundary: when a team is reproducing a known training recipe to validate a new parallelism or hardware stack, accelerator utilisation can be the primary success measure because learning quality is deliberately held constant. Once the run chooses new data, a new mixture, or a larger model, that exception ends; throughput alone no longer describes the value of scale.

My preferred operating rule is simple: approve compute in stages, and require each stage to earn the next through measured marginal learning. The rule joins infrastructure efficiency to evidence about model improvement instead of letting one stand in for the other. Accelerators are rented capacity. The durable asset is the evidence about which data, in which sequence and mixture, continues to improve the model. I would scale the cluster only after that evidence shows there is still productive learning for the cluster to accelerate.