Data Multiplicity Is a Training Measure

1. Repetition changes the question being optimized

A training corpus is a measure, not merely a set of strings. If the same observation occurs many times, sampling rows uniformly gives that observation more influence. Removing duplicates changes those weights unless the optimization algorithm restores them deliberately. The resulting model may improve, deteriorate, or simply answer a different population question. The distinction matters whenever deduplication is described as a harmless storage optimization.

This review separates record identity, sampling probability, population representation, and optimization exposure. Our finite construction uses three scalar observations and weighted squared loss, so every optimum and effective sample count can be checked without a neural training run. Its simplicity exposes an accounting error that remains possible in much larger pipelines: comparing two datasets with different multiplicities while calling their objectives identical. The scientific contribution here is a transparent analytical comparison and a proposed evaluation design. It is not a new empirical measurement of language-model memorization, privacy, or downstream accuracy. Those outcomes require task-specific experiments with their own data and uncertainty.

2. What six primary records establish

Lee and colleagues study exact and approximate text deduplication, including memorized output and train-test overlap. Kandpal and colleagues connect repetition with extraction risk under studied attacks. Abbas and colleagues use embedding similarity for semantic deduplication. These are different operational notions of redundancy, rather than a single universal duplicate relation.

DataComp provides a controlled framework for studying multimodal dataset choices. Sorscher and colleagues analyze data pruning and its dependence on the quality of a ranking metric. Muennighoff and colleagues examine repeated exposure under constrained language-model data and compute. Their scopes motivate separate controls for content selection, multiplicity, and total training budget. We use their primary abstracts to establish these research questions; we do not transfer their reported improvements to another corpus. In particular, successful repetition in one resource regime does not imply that accidental duplication is beneficial, and a reduction in a particular extraction attack does not constitute a differential privacy guarantee. No source experiment is reproduced in this review.

Sources: [1], [2], [3], [4], [5], [6]

3. Define the unit before counting duplicates

An exact byte hash identifies repeated files, but preprocessing can map several files to the same token sequence. Conversely, visually similar images can encode distinct labels, views, times, or rights. A duplicate relation must specify the representation at which identity is tested. Document, paragraph, sentence, image, and training-token units lead to different retained collections.

For near duplicates, pairwise similarity need not be transitive. If A resembles B and B resembles C, A need not resemble C under the same threshold. Taking connected components imposes an additional closure operation, potentially merging a long chain of genuinely different observations. The representative-selection rule then determines whose language, timestamp, resolution, or annotation survives. Reporting only a removal percentage conceals both operations. A reproducible corpus release should describe normalization, similarity metric, threshold, clustering rule, representative selection, and ordering. It should also distinguish intentional repeated measurements from accidental copies. Ten independently measured outcomes with the same value are not equivalent to ten copies of one measurement; content equality alone cannot recover that provenance.

4. Weighted empirical risk makes the change explicit

Let unique observations be z_i with positive integer multiplicities m_i and total row count M. Uniform row sampling minimizes the weighted empirical risk sum_i m_i loss(theta,z_i)/M. Retaining one representative per exact class instead gives weight 1/K to each of K unique observations. The two risks coincide only under restrictive conditions, such as equal multiplicities, or when different weights happen to yield the same loss landscape.

For scalar squared loss, the optimum is the weighted mean. Our examples are z=(-1,0,2) with multiplicities (8,1,1), giving theta=-0.6. Uniform unique-observation weighting gives theta=1/3. Both are mathematically correct minimizers of their declared risks. Calling one the true optimum requires a target distribution that is not supplied by the file inventory. If multiplicities represent real prevalence, deduplication can remove useful population information. If they reflect scraping artifacts, retaining them can exaggerate irrelevant frequency. The central empirical question is which interpretation applies, not whether repeated rows are inherently good or bad.

5. A distortion curve separates population and collection

Hold the desired target distribution uniform over the three observations and increase the multiplicity of the negative observation from one through sixteen. The row-weighted optimum becomes (2-m)/(m+2), while the target optimum stays 1/3. With half squared loss, the excess target risk is one half of their squared difference. This identity follows by expanding the quadratic around its mean, so the figure is fully determined by the declared construction.

The curve is an illustration of accidental oversampling under this particular target, not a fitted scaling law for deep learning. Reversing the target assumptions can reverse the recommendation: if the negative observation truly becomes more prevalent, its increased training weight may be appropriate. We therefore display both the learned mean and the excess risk against a fixed explicit target. A plot containing only the training objective would favor each dataset's own optimum and fail to show the disagreement. Evaluation is meaningful only when the quantity being compared remains fixed while the intervention changes.

6. Effective sample counts describe concentration, not independence

For normalized weights w_i, the quantity 1/sum_i w_i squared is often used to summarize weight concentration. With multiplicities (8,1,1), it equals 100/66, about 1.52, despite ten stored rows and three unique values. After uniform deduplication it equals three. This is an exact arithmetic description of concentration across the chosen unique classes.

It is not automatically the number of independent observations available for generalization. The three retained values might themselves come from one source, while repeated equal values could have been independently measured. Dependence, cluster structure, and the estimator's variance matter. The second construction varies the dominant multiplicity and contrasts stored rows, unique classes, and this weight-based quantity. Their divergence makes an overloaded dataset-size claim visible. A report should name the denominator used in learning curves: bytes, tokens, rows, unique documents, sources, or independent sampling units. Each measures something different, and replacing one with another after inspecting results can create an apparent efficiency gain without adding evidence.

7. Exact compression can preserve the objective

Removing physical copies does not require discarding their statistical weights. A compact representation can retain each unique example together with multiplicity and sample it proportionally. For deterministic full-batch loss evaluation, this exactly preserves the original objective. With stochastic training, preserving the expected gradient still leaves questions about update order, batch composition, augmentation randomness, and optimizer state.

For example, two identical images augmented independently are not necessarily identical training inputs. Collapsing them into one cached augmented tensor changes the distribution of views even when the nominal multiplicity weight is retained. Likewise, gradient accumulation followed by one adaptive-optimizer update is generally different from several sequential updates on the repeated example. The appropriate equivalence claim should specify whether it concerns the mathematical risk, expected gradient, complete stochastic trajectory, or final model distribution. Compression can provide storage and I/O benefits without being an experimental deduplication policy. Keeping these mechanisms separate allows one to attribute an observed benefit to its actual cause.

8. Semantic pruning adds a representation-dependent judgment

A semantic filter compares examples through an embedding model. Its geometry inherits that model's training distribution and invariances. Two observations that are close in the embedding may differ in precisely the attribute needed for a downstream task. A threshold that removes redundant product photographs could also erase rare defects, dialectal distinctions, or subtle scientific conditions.

This does not invalidate semantic pruning. It means that similarity is a task-conditioned decision variable. One can test it using labeled disagreement pairs, rare-category retention, and stability across independently trained embeddings. The evaluation should include examples close to the threshold, since those reveal how fragile the selected set is. A representative selected by quality score may systematically favor high-resolution or majority-language content. That is a second intervention beyond merging. To understand a model improvement, report the joint effect of cluster construction and representative choice, together with sensitivity to both. A single overall retention fraction cannot explain which information was actually preserved.

9. Train-test leakage requires a directional policy

Deduplicating the training set internally does not establish independence from evaluation data. An evaluation item can have one exact or transformed match in training and still contaminate the comparison. Conversely, removing every training item semantically close to a test example can alter the task by making the training distribution artificially distant from the evaluation distribution.

The policy should identify whether the objective is preventing exact leakage, template leakage, source overlap, or shared latent events. Split by the highest relevant provenance unit before threshold tuning, then use a separate development collection to choose the deduplication mechanism. Test-set information should not silently influence which training corpus wins. When benchmarks are public, report known overlap and uncertainty instead of claiming impossible proof of absence. Content matching has false negatives under paraphrase and false positives for generic phrases. Both error types have consequences, so a leakage audit should sample suspected matches and unmatched examples with an explicit review protocol rather than relying on one opaque score.

10. Budget-matched controls distinguish three explanations

A deduplicated model can consume fewer unique rows, fewer total tokens, or the same tokens through more epochs. These comparisons answer different questions. Under a fixed epoch count, pruning usually lowers training compute; under a fixed token budget, it increases repeated exposure to survivors; under a fixed wall-clock budget, pipeline throughput and preprocessing costs also matter.

A useful experiment therefore has separate budget axes. First compare objectives at matched total updates and matched token counts. Then compare practical time including deduplication, feature extraction, indexing, and storage. Finally compare target risk at a fixed independently selected stopping rule. Report the complete budget rather than naming a single favored accounting unit. If the retained examples are shorter, token-budget matching may change document exposure even with equal row counts. These are not minor implementation details: they determine the number and type of gradient contributions. The constructed weighted-mean example isolates only the objective component, providing a clean baseline against which these additional effects can be discussed.

11. Rare information can be repeated for legitimate reasons

The relationship between rarity and duplication is not one-directional. A rare language may have only a few accessible documents copied across many websites. Removing copies reduces its row count, while keeping them can exaggerate the importance of a small number of authors. Neither choice creates missing linguistic diversity. Retaining one copy with explicit domain balancing can be a defensible third policy.

Similarly, a medical or industrial record may contain repeated templates with a small but essential variable field. Whole-document similarity can suppress the field along with boilerplate. The unit of analysis should reflect the scientific information: perhaps template text should be compressed while measurements retain provenance and sampling weights. Such policies need error analysis at the level of meaningful variation. A high overall compression ratio cannot establish preservation of rare outcomes. Report category coverage, source diversity, and annotation disagreement before and after selection, including categories for which there are too few independent examples to estimate a reliable effect.

12. Auditability is part of the dataset intervention

A deduplication manifest should preserve stable identifiers for original records, equivalence classes, retained representatives, and policy version. Where content cannot be redistributed, approved nonreversible identifiers and aggregate diagnostics can still document what operation occurred. The manifest should allow an authorized reviewer to trace a removal decision without exposing private raw examples.

Threshold changes must produce a new version rather than silently modifying the previous dataset. Model comparisons need both the original and selected manifests, because a downstream change in tokenization can invalidate old duplicate boundaries. Recording only the random seed is insufficient when clustering depends on nondeterministic approximate-neighbor search or unstable input order. A reproducible pipeline should test whether reordering the same inputs changes representatives and whether such changes alter category balance. These are practical scientific controls, not administrative overhead. They establish which training measure produced the reported model and enable a later correction without rewriting the historical experiment.

13. A discriminating evaluation plan

Begin with a corpus whose provenance groups and evaluation splits are frozen. Register exact, near-text, and embedding-based duplicate definitions separately. Sample false merges and missed matches, then estimate their frequency with uncertainty and category breakdowns. Train matched baselines using full multiplicity, unique uniform weights, and unique multiplicity-preserving weights. The third arm tests whether physical compression alone explains performance.

Use both fixed compute and fixed unique-content budgets, reporting tokenization and augmentation policies. Evaluate on an untouched target set with source-level bootstrap intervals when sources are the independent units. Include a distinct extraction-risk evaluation if privacy is a claim; downstream accuracy cannot stand in for it. Preserve every failed run and report filter preprocessing costs. The expected outcome is a map of tradeoffs, not necessarily one winning policy. If unique-uniform weighting improves the target metric but harms a rare domain, that is an informative distributional result requiring a deployment choice, rather than a nuisance to hide in the average.

14. What the finite checks can and cannot establish

The accompanying program enumerates multiplicities, computes weighted optima, and checks the quadratic excess-risk identity against direct loss evaluation. It also verifies that equal multiplicities recover the uniform unique optimum and that multiplying every multiplicity by a common factor leaves the weighted objective unchanged. These negative and invariance controls detect denominator mistakes in the figures.

They do not test a semantic embedding, a neural optimizer, memorized output, or a real corpus. Their role is to make the accounting logic falsifiable with exact small objects. The broader research opportunity is to connect this transparent accounting with task-sensitive duplicate relations and measured learning behavior. A scientifically defensible conclusion should state which measure was changed, why that measure better matches the intended population, and what information might have been lost. Deduplication is then understood as an explicit statistical intervention whose value can be tested, rather than a generic promise that fewer rows necessarily contain the same science.

15. An exact sensitivity calculation for a mistaken merge

The most consequential failure of a duplicate detector may be a false merge rather than a missed duplicate. Consider two distinct scalar observations a and b that are collapsed into the representative a. Under uniform weighting of the original two observations, the squared-loss optimum is (a+b)/2. After retaining only a, the optimum is a. If the intended target remains the original two-point distribution, the excess half squared risk is (a-b) squared divided by eight. The effect grows quadratically with the information lost in the merge, even though the storage reduction is always fifty percent.

This small calculation illustrates why a compression ratio has little meaning without a false-merge cost model. A detector could remove many harmless repeated boilerplate strings and a few scientifically essential distinctions, producing an attractive aggregate number while damaging the downstream objective. The relative importance of the distinctions is task-dependent. In a classification problem, two near-identical images with different labels may indicate annotation noise, a subtle relevant feature, or an ambiguous taxonomy. Automatic similarity cannot decide among those explanations without additional evidence.

A practical audit can use an adjudicated sample of proposed merges, stratified by similarity score and category. Estimate not only how often a merge is questionable, but how much its alternatives could change the intended target. For regression, label or measurement differences provide one useful diagnostic. For language, preserving named entities, quantities, negation, and time references may require separate checks. These are proposed controls, not results of an audit performed here.

The executable model includes the exact two-point false-merge identity as an additional check, independent of the three-point multiplicity sweep. It verifies the zero-cost limit a=b and symmetry under exchanging the two points. Together the constructions distinguish accidental overweighting from accidental loss of distinct information. A good policy may need to address both, and optimizing one alone can worsen the other.

Core equations and definitions

Constructed analytical figures

Comparison and reporting contract

Evaluation protocol

• Freeze provenance and test splits before filtering.
• Compare unique uniform and multiplicity-preserving arms.
• Match both token and practical compute budgets.
• Audit false merges and rare-category retention.

Limitations

• No actual corpus is deduplicated.
• Scalar squared loss does not model neural memorization.
• The target distribution is explicitly constructed.

Open research questions

• How should duplicate similarity reflect task-relevant distinctions?
• How can rare-domain retention be audited efficiently?
• Which budget best predicts practical benefit?

Evidence and its boundaries

Exact and approximate deduplication affect memorized output and train-test overlap in the studied language-model settings.

[1] — Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.

Repetition is associated with extraction risk under the evaluated attacks.

[2] — Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.

Embedding-based semantic deduplication is a distinct data-selection mechanism.

[3] — Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.

A controlled testbed studies dataset choices under standardized multimodal training.

[4] — Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.

Data-pruning benefits depend on the ranking metric and studied setting.

[5] — Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.

Repeated exposure has a compute- and data-regime-dependent value.

[6] — Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.