Deduplicating training data is capacity allocation, not housekeeping

The first deduplication pass removes the obvious duplicates — identical text across pages that scraped the same article, code examples copied verbatim into multiple repositories. You get a measurable reduction in corpus size and a bounded increase in per-token loss from training on fewer tokens.

A second round removes near-matches — articles by different authors covering the same event with overlapping language, code blocks repeated with minor formatting changes across tutorials. The corpus shrinks further; each token carries higher unique information density. But it also costs more compute to find.

This allocation problem is what separates deduplication from housekeeping. If you stop early (cheap dedup), you waste fewer tokens but preserve some redundancy. Stop late (expensive dedup), you maximize uniqueness at the cost of search time and memory. The right stopping point depends on corpus size, compute budget, and — critically — how much value the model gets from seeing overlapping data before adding marginal signal.

Capacity allocation as a resource management problem

Every dedup round spends FLOPs to save tokens. Early rounds spend less compute and recover more unique data; later rounds spend more per saved token. The total savings compound, but at decreasing margins — each round extracts less new value from the same corpus.

This means dedup is not a linear operation. If you have enough tokens to deduplicate thoroughly, you spend compute to remove redundancy and save training cycles. But if corpus size grows faster than your ability to deduplicate, you can reach saturation (too much volume for any single pass) — where total corpus size outpaces dedup efficiency gains.

The trade surface

Early stop: cheap compute spend, less total unique data. Late stop: more compute search overhead, higher unique token density. For large corpora (billions of tokens), early dedup is usually better because the marginal information gain from expensive operations drops below the cost of finding them.

The choice is real: your corpus and compute budget define a frontier. You can maximize dedup thoroughness or minimize search spend — not both simultaneously in every regime. The frontier depends on corpus size, available compute for dedup, and what kind of model you are training next.

The implication for agent-era data

If the training corpus is expanding faster than dedup operations can run, the frontier compresses: fewer dedup passes are feasible relative to total unique tokens retained. You either accept lower curation or slow down training throughput. This is not a housekeeping decision — it is an architecture choice between data volume and data quality as your primary optimization vector.