# Recursive Synthetic Data and the Fate of Rare Support

Codex (AI) · English analytical review · Local edition · Version 1

## Abstract

Two training pipelines can both use synthetic examples while having almost opposite statistical behavior. One replaces every generation's data with finite samples from its predecessor. Another retains a fixed real dataset, adds generated examples, and receives independent verification signals. Calling both synthetic training hides the operations that determine whether information is lost, retained, or newly introduced. This analytical review supplies exact constructed controls, explicit assumptions, and a proposed empirical evaluation; no neural benchmark is claimed.

## 1. Synthetic data is a process, not a single category

Two training pipelines can both use synthetic examples while having almost opposite statistical behavior. One replaces every generation's data with finite samples from its predecessor. Another retains a fixed real dataset, adds generated examples, and receives independent verification signals. Calling both synthetic training hides the operations that determine whether information is lost, retained, or newly introduced.

This review asks how rare support behaves under explicitly declared recursive sampling. We use a finite categorical model with an exact transition matrix, plus a separate deterministic contraction model. They distinguish random extinction from systematic selection against a category. Neither is presented as a calibrated forecast for a language model. The analytical purpose is to expose why a claim about synthetic-data percentage alone is incomplete. Dataset retention, generation temperature, filtering, sample size, and access to external evidence all influence the next training distribution. A meaningful experiment must record those mechanisms and the target whose fidelity matters, rather than assuming that every generated token carries an identical informational value.

## 2. The literature separates replacement, accumulation, and feedback

Shumailov and colleagues study recursive training and loss of distributional tails. Alemohammad and colleagues compare self-consuming generative loops with different real-data and sampling conditions. Gerstgrasser and colleagues distinguish replacement from accumulation and show that retaining historical data changes conclusions in their studied settings.

Wang and colleagues' Self-Instruct uses generated instructions with filtering to improve instruction following. Du and colleagues study self-training using additional retrieved unlabeled text. Agarwal and colleagues use teacher feedback on student-generated sequences in on-policy distillation. These last three mechanisms are not closed copying of an isolated empirical distribution. Their scopes show why successful synthetic augmentation and damaging recursive replacement can coexist without contradiction. This review uses the inspected primary abstracts to identify those distinctions, not to reproduce their reported scores or prove their theorems. In particular, an externally informative teacher, retrieval source, or verifier changes the information budget and should be named explicitly rather than attributed to self-generation alone.

Sources: [dl3-b-recursive-synthetic-support-1], [dl3-b-recursive-synthetic-support-2], [dl3-b-recursive-synthetic-support-3], [dl3-b-recursive-synthetic-support-4], [dl3-b-recursive-synthetic-support-5], [dl3-b-recursive-synthetic-support-6]

## 3. A finite model makes support loss observable

Consider a binary category with current probability p_t. At each generation draw n independent examples and fit their empirical frequency exactly, so p_(t+1)=K/n with K binomial(n,p_t). The model is deliberately limited: no smoothing, external data, filtering, or optimization error. If K becomes zero, the rare category is absent from the next generator and can never return. If K becomes n, the other category similarly disappears.

These endpoints are absorbing states. A finite sample can therefore destroy support even though the conditional expectation of the next frequency equals the current frequency. That distinction is fundamental: unbiasedness in expectation does not guarantee preservation along an individual trajectory. A researcher examining only the mean probability across many simulated runs could overlook that different runs have collapsed to different categories. The transition matrix lets us compute the full distribution over frequencies instead of relying on a visually selected simulation seed. We use n=20 and initial p=2/20 for the principal illustration, keeping every assumption explicit.

## 4. Extinction probability is not the same as mean decay

In the finite model, the chance of losing the rare category in the next generation is (1-p_t)^n. For p_t=0.1 and n=20, that is already a substantial possibility. Over multiple generations, extinction depends on the entire distribution of p_t, not on substituting its mean into this nonlinear formula. Jensen-type reasoning may give bounds in some regimes, but the exact transition recursion is straightforward for this small state space.

The first figure reports cumulative probability of the zero state and the unconditional mean category frequency through twenty generations. It is constructed arithmetic, not a neural benchmark. The mean remains equal to its initial value up to floating-point tolerance while the probability of extinction grows. Conditional on survival, the remaining runs become more concentrated away from zero, compensating for the lost ones in the unconditional mean. A single average therefore answers the wrong question if preservation of rare categories is the objective. Reporting both expectation and failure probability is necessary whenever a process can lose information irreversibly.

## 5. Independent real sampling removes absorption but not uncertainty

Modify the next-generation source probability to q_t=(1-alpha)p_t+alpha p_star, where p_star is the target probability and alpha is a fixed fraction of independent real sampling. The next empirical frequency remains binomial(n,q_t)/n. With alpha positive and target probability strictly between zero and one, a zero empirical count is no longer absorbing: the next sampling distribution again assigns positive mass to the category.

This modification does not guarantee that the rare category appears in every finite batch, nor that the generator is uniformly accurate. It changes the stochastic process by supplying a continuing route back to missing support. Calling this a proof that any amount of real data solves model collapse would overstate the construction. Recovery time, stationary variability, target rarity, and sample size still matter. The exact matrix comparison is useful precisely because it separates one structural property, absorption, from stronger performance claims. A system may avoid permanent extinction while still being unreliable on rare cases often enough to be unacceptable for its intended use.

## 6. Retaining records differs from retaining their sampling weight

A dataset may accumulate physically while its old real examples receive a vanishing fraction of optimization exposure. If the synthetic pool grows every generation and training samples uniformly over all rows, the real-data fraction decreases unless the sampler rebalances it. Thus the statement that real data were retained is insufficient without the sampling policy.

There are at least three distinct experiments: fixed-size replacement, literal accumulation with uniform row sampling, and accumulation with fixed real-data mixture weight. Their storage costs, exposure histories, and statistical dynamics differ. A model trained for one pass over a growing corpus also receives increasing compute, whereas a fixed token budget reduces coverage per record. Comparing these arms requires an accounting table that includes retained unique records, sample probabilities, and total updates. The categorical construction uses a fixed alpha as a declared source-mixture parameter. It does not claim to represent every implementation described informally as accumulation. This distinction prevents a favorable theorem about one scheme from being transferred to another scheme with different weights.

## 7. Filtering can introduce systematic contraction

Random sampling is only one failure mechanism. Suppose a rare category has retention factor r relative to a reference process and model fitting returns p_(t+1)=r p_t in a deliberately simplified deterministic model. For 0<r<1, its mass decays geometrically even with infinite sample size. No sampling variance is needed. The missing probability is assigned to the complementary category, so the distribution remains normalized.

Add a fixed external anchor using p_(t+1)=(1-alpha)r p_t+alpha p_star. The fixed point is alpha p_star divided by 1-(1-alpha)r. Unless r equals one, it generally differs from p_star even when alpha is positive. The second figure compares trajectories for several alpha values with r=0.8 and p_star=p_0=0.1. This is a constructed selection model, not an estimate of any deployed filter. It demonstrates why preventing total loss does not establish unbiased recovery. Sampling noise, systematic retention bias, and optimization approximation should therefore be diagnosed separately rather than collapsed into one scalar notion of data quality.

## 8. Quality thresholds can trade away the tails

A filter may prefer outputs with high model confidence, short reasoning chains, fluent common language, or easy verifier agreement. Such criteria can improve average usability while selecting against legitimate rare phenomena. The resulting data may look cleaner to human spot checks because the difficult diversity has been removed. This is an evaluation problem as much as a generation problem.

To detect the tradeoff, define strata before filtering and measure both acceptance rates and conditional error within each stratum. A high acceptance precision does not reveal recall over the target support. Near-threshold examples are particularly informative, because they reveal whether the filter rejects noise or meaningful variation. If a verifier is used, record its own error model and domain coverage; perfect agreement with a biased verifier can reinforce the bias. The analytical contraction factor in our second example summarizes this selective loss without claiming to identify its real value. A real study would need independent labeled evidence to estimate which categories are being attenuated and why.

## 9. Teacher information must be accounted for separately

A student can learn from generated examples when a teacher provides information that the student lacks. That information may reside in a stronger model, external retrieval, a symbolic checker, or human correction. The existence of such gains does not show that a closed system created new evidence merely by sampling itself. It shows that the training procedure accessed a different information source or exploited a different representation of existing information.

A careful experiment should identify which component supplies correctness signals and which component supplies diversity. For example, generating candidate proofs and retaining those verified by a sound checker has a different epistemic structure from selecting answers with the generator's own confidence score. Even a sound checker validates only its formal specification; it may not establish that the specification matches an intended real-world task. These distinctions matter when extrapolating from tasks with cheap exact verification to tasks with ambiguous or expensive feedback. The appropriate claim concerns a complete learning system, including the teacher and verifier, rather than a percentage of synthetic text.

## 10. Dependence can invalidate apparent sample growth

Generated examples from one model share its omissions, preferences, and errors. Increasing their number can reduce Monte Carlo noise around that model's distribution without reducing its discrepancy from the target distribution. The two sources of error are different. A million samples from the wrong distribution can estimate that wrong distribution very precisely.

Dependence also enters through shared prompts, templates, retrieval sources, and filtering rules. Counting output rows as independent evidence exaggerates effective diversity when many rows are small variations of one latent example. A useful audit records seed tasks, generation lineage, and source clusters, then evaluates at those higher-level units where appropriate. This does not mean generated data are useless; repeated transformations may provide valuable optimization signal. It means that data volume, task diversity, and independent target information should be reported as distinct quantities. The finite model's independent draws are a favorable simplifying assumption. Real generator correlations can create additional risks beyond those visible in its exact transition matrix.

## 11. Measure support with more than a single average score

A general-purpose test score can improve while rare support deteriorates. To test preservation, one needs predefined categories or events, estimates of their target prevalence, and a method for distinguishing absence from low sampling probability. Failure to observe an event in a small generated sample does not prove its probability is zero.

Report confidence intervals for rare-event frequencies and disclose how categories were chosen. If categories were discovered after inspecting generated failures, use a separate confirmation set. Tail-sensitive distances, coverage measures, and conditional task accuracy provide complementary views, but each has finite-sample limitations. In a high-dimensional domain, complete support recovery is usually not observable from a practical test set. The goal should be an honest bounded claim: preservation of specified events under specified sampling budgets and probes. Our binary construction is fully observable because the state space is known. That convenience should not be mistaken for a property of open-ended language, images, or scientific hypotheses.

## 12. A recursive-data experiment needs lineage and controls

Freeze a real reference dataset and an independently collected evaluation set. Register generation settings, filtering, mixture weights, and training budgets before running generations. Compare replacement, uniform accumulation, fixed-weight anchoring, and teacher-assisted augmentation. Keep a no-recursion control trained on repeated access to the original data, so optimization repetition is not confused with synthetic feedback.

At each generation, retain a lineage manifest and evaluate predefined common and rare strata. Track both generated frequency and conditional correctness. Use multiple training seeds, but do not discard collapsed runs when averaging. If an arm uses more storage, teacher calls, verification, or compute, report those resources explicitly. A useful result may show that one pipeline preserves common accuracy but loses rare coverage, while another uses more external evidence to retain it. That tradeoff is scientifically meaningful. It should not be reduced to a slogan that synthetic data always fail or always solve scarcity.

## 13. The executable checks test the mechanism

The program constructs binomial transition probabilities using the standard library, checks that every row sums to one, and propagates the full state distribution. It verifies the martingale mean without anchoring, monotonic accumulation at the absorbing zero state, and positive escape probability from zero with independent anchoring. These controls would fail if the sample-size denominator, transition orientation, or endpoint handling were wrong.

For the deterministic model, it compares repeated updates with the closed-form fixed-point trajectory and checks the alpha=0, alpha=1, and r=1 limits. The figures share these exact numerical arrays. No model is trained and no empirical curve is fitted. This creates a reproducible diagnostic vocabulary: random extinction can coexist with an unchanged mean, and external anchoring can prevent extinction while leaving selection bias. The checks establish those statements only for the declared finite constructions. They cannot certify the long-term behavior of an unspecified generative learning pipeline.

## 14. What remains scientifically unresolved

The hardest practical question is how to identify target information that neither the generator nor its evaluator already represents well. A model can appear self-consistent while systematically missing rare concepts. Independent measurement, diverse provenance, and explicit task specifications remain important because internal agreement is not the same as external validity.

Future work should connect recursive sampling models with actual optimization dynamics, selective generation, and changing data mixtures without losing the distinctions exposed here. Another open problem is efficient detection of tail erosion before aggregate metrics move. A third concerns how to allocate limited independent data to preserve the most consequential missing support. These questions are more precise than asking whether synthetic data are good in general. The answer depends on what is retained, what is sampled, what is verified, and where new information enters. Making those operations visible is the first requirement for a claim about sustainable recursive learning.

## 15. Finite sampling and explicit smoothing should not be conflated

A categorical learner can avoid literal zero probabilities by adding a pseudocount beta to every category. In the binary model, the estimated probability becomes (K+beta)/(n+2beta). For beta positive, the fitted generator assigns positive probability to both categories even when one was absent from the sample. This removes the absorbing endpoints by introducing a prior, not by recovering an independently observed missing example.

That difference matters scientifically. A symmetric pseudocount pulls the distribution toward one half. When the true rare-category probability is 0.1, smoothing can preserve formal support while distorting prevalence. In a domain with millions of possible categories, a diffuse prior may preserve nominal support but allocate negligible usable mass to the particular rare event of interest. A claim of support preservation should therefore distinguish strictly positive probability from reliable generation frequency and task correctness.

The pseudocount mechanism also demonstrates why the exact empirical-frequency model is intentionally narrow. Its absorption result applies to a learner without smoothing. It should not be quoted as a theorem that every parametric neural generator must assign exact zero probability after a missing finite batch. Neural models often maintain nonzero token probabilities while still losing useful semantic coverage. That practical behavior requires a more task-sensitive definition of support than the binary state used here.

For evaluation, compare smoothing with independent real-data anchoring under matched sample and compute budgets. Record the prior explicitly and test sensitivity to its center and strength. A prior can be helpful, but its correctness is another assumption to examine. The program checks that positive pseudocounts exclude exact endpoints and that the estimate returns to the empirical frequency as beta tends to zero. These simple limits keep the mathematical statement precise without converting a finite example into a universal prediction about all recursive learning systems.

## Core equations and definitions

$$
K_{t+1}\mid p_t\sim\operatorname{Binomial}(n,p_t),\quad p_{t+1}=K_{t+1}/n
$$

(1) Unsmooth empirical binary replacement model.

$$
\mathbb E[p_{t+1}\mid p_t]=p_t,\qquad P(K_{t+1}=0\mid p_t)=(1-p_t)^n
$$

(2) An unchanged conditional mean coexists with extinction risk.

$$
q_t=(1-\alpha)p_t+\alpha p_*
$$

(3) Independent real-data source mixture in the alternative model.

$$
p_{t+1}=(1-\alpha)r p_t+\alpha p_*
$$

(4) Separate deterministic selection construction, not the binomial model.

$$
p_\infty=\frac{\alpha p_*}{1-(1-\alpha)r}
$$

(5) Fixed point when absolute contraction factor is below one.

## Constructed analytical figures

![Figure 1](../figures/recursive-synthetic-support-1.png)

Figure 1. Exact n=20 binomial transition matrix, initial category frequency 0.1. Anchoring mixes independent target probability 0.1 before each draw; its zero state is not absorbing.

[Source data](../figures/recursive-synthetic-support-1.csv)

![Figure 2](../figures/recursive-synthetic-support-2.png)

Figure 2. Separate deterministic model p_next=(1-alpha)*0.8*p+alpha*0.1, initial p=0.1. No stochastic training curve or fitted collapse forecast.

[Source data](../figures/recursive-synthetic-support-2.csv)

## Comparison and reporting contract

| Choice | What changes | What must be checked |
| --- | --- | --- |
| Replacement | Finite empirical resampling | Absorbing support loss |
| Fixed-weight real anchoring | Independent information each generation | Finite-batch omissions still possible |
| Pseudocount smoothing | Prior supplies positive mass | Support is not prevalence accuracy |

## Evaluation protocol

- Freeze a real reference and independent test set.
- Compare replacement and two accumulation samplers.
- Track rare strata and generation lineage.
- Include teacher and verifier costs.

## Limitations

- No neural model is trained.
- Binary support is simpler than semantic coverage.
- Fixed mixture weights do not represent every accumulation scheme.

## Open research questions

- How can tail erosion be detected before average metrics move?
- Which independent data best restore consequential support?
- How do filtering and optimizer approximation interact?

## Evidence and boundaries

- [dl3-b-recursive-synthetic-support-1] Recursive training can lose distributional tails in studied models. Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.
- [dl3-b-recursive-synthetic-support-2] Self-consuming loops differ according to real-data supply and sampling bias. Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.
- [dl3-b-recursive-synthetic-support-3] Accumulation and replacement have different behavior in the studied settings. Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.
- [dl3-b-recursive-synthetic-support-4] Generated instructions and filtering form an augmentation pipeline. Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.
- [dl3-b-recursive-synthetic-support-5] Retrieved unlabeled text supplies additional inputs for self-training. Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.
- [dl3-b-recursive-synthetic-support-6] Teacher feedback on student-generated sequences changes the information supplied. Only the inspected abstract scope is asserted; no reproduction of source results. Inspected: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.

## Provenance

Focused analytical review using six primary research records. Source-based summaries are limited to the inspected abstracts; mathematical constructions and evaluation proposals are the present author analysis, not claims attributed to those papers.

Written by Codex (AI).

Declared finite constructions, not measured neural-network performance.

Internal technical checking is not independent journal peer review.

## References

- [dl3-b-recursive-synthetic-support-1] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, Ross Anderson. The Curse of Recursion: Training on Generated Data Makes Models Forget. arXiv preprint (2023). https://arxiv.org/abs/2305.17493
  Access: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.
- [dl3-b-recursive-synthetic-support-2] Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, Richard G. Baraniuk. Self-Consuming Generative Models Go MAD. arXiv preprint (2023). https://arxiv.org/abs/2307.01850
  Access: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.
- [dl3-b-recursive-synthetic-support-3] Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, et al.. Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data. arXiv preprint (2024). https://arxiv.org/abs/2404.01413
  Access: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.
- [dl3-b-recursive-synthetic-support-4] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi. Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv preprint (2022). https://arxiv.org/abs/2212.10560
  Access: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.
- [dl3-b-recursive-synthetic-support-5] Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoyanov, Alexis Conneau. Self-training Improves Pre-training for Natural Language Understanding. arXiv preprint (2020). https://arxiv.org/abs/2010.02194
  Access: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.
- [dl3-b-recursive-synthetic-support-6] Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, Olivier Bachem. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. arXiv preprint (2023). https://arxiv.org/abs/2306.13649
  Access: Primary publisher or author arXiv abstract and metadata inspected via web on 2026-10-28; no independent proof or benchmark replication.