What a Mask Forgets: Discrete Diffusion, Joint Dependence, and Parallel Token Generation

1. Tokens require an explicit corruption law

A categorical token does not have a canonical infinitesimal neighborhood. Replacing one word identifier with the next integer does not define a meaningful small perturbation. A discrete diffusion model must therefore specify a transition rule that describes how a token or sequence is corrupted. The rule can erase information, replace a symbol, move between selected neighbors or use another structured state space.

This choice affects the reverse task. A masked symbol announces that information was removed, while a randomly substituted ordinary symbol can be mistaken for uncorrupted content unless the model uses the correct probabilistic interpretation. The forward rule also determines the distribution from which generation starts and the conditions under which a reverse transition is defined. We focus on these contracts rather than treating all masked or non-autoregressive models as one method. The finite examples below isolate kernel semantics and joint dependence, with no claim to reproduce a trained language or image generator.

2. Structured transitions create different denoising problems

Austin and colleagues formulate discrete denoising diffusion with transition matrices beyond uniform replacement, including structured and absorbing choices. This provides a principled vocabulary for comparing corruption processes. The matrix is not merely an implementation detail: it determines which distinctions remain visible at a given time and which reverse predictions require inference from context.

For a finite state space, a row-stochastic matrix maps a distribution to another distribution under a chosen row-vector convention. Every row must contain nonnegative values and sum to one. Composing time steps multiplies the matrices in the same convention. Confusing row and column conventions can yield attractive-looking equations with the wrong transition law. A minimal implementation test should verify normalization, nonnegativity, composition and the declared limiting distribution. It should also include special symbols such as a mask in the state space when they participate in transitions. These checks validate the forward construction before any neural-network approximation enters the problem.

Sources: [1]

3. Absorbing masking and replacement are not equivalent

Consider two ordinary symbols and an additional absorbing mask. At each step, an ordinary symbol is retained with probability alpha and otherwise becomes masked; a mask stays masked. After several steps, the probability of still observing the original symbol is the product of the retention probabilities. Once erased, that symbol is not restored by the forward process.

Now consider uniform replacement over the two ordinary symbols, using a matrix equal to alpha times the identity plus one minus alpha times the uniform kernel. After a composed retention factor r, the probability of observing the original symbol is r plus half of one minus r. This includes accidental replacement back to the same symbol. At r equal to 0.4, absorbing masking leaves the original visible with probability 0.4, while replacement shows the original with probability 0.7. The difference does not mean replacement preserved more usable information by that amount. At complete replacement, the observed symbol matches the original half the time by chance. An evaluation must distinguish visible equality from information retained about the original.

4. A two-step kernel is independently checkable

Set the first retention probability to 0.8 and the second to 0.5. Their product is 0.4. An explicit matrix multiplication for the absorbing process gives probability 0.4 on the original symbol and 0.6 on the mask. For uniform replacement over two symbols, the same composed parameter gives probability 0.7 on the original and 0.3 on the other symbol. These values can be enumerated from the intermediate states as a separate check.

The calculation is deliberately small enough to expose implementation errors. A script should compare direct multiplication with path enumeration, confirm that the absorbing state never leaves itself and verify the replacement kernel's limiting behavior. It should deliberately reject a matrix with a negative entry or a row whose probabilities do not sum to one. These tests do not establish that a learned reverse model is accurate. They establish which forward problem that model is being asked to invert, and prevent a mislabeled corruption schedule from contaminating every later experiment.

5. Marginal predictions can miss a joint constraint

Let a target distribution put equal mass on the two sequences 00 and 11. Each position separately has a uniform binary marginal. If a reverse model begins with both positions masked and samples the two predictions independently from those marginals in a single round, it produces all four binary sequences equally. Half of the results, 01 and 10, are absent from the target distribution.

This is not an impossibility theorem for parallel generation. The conclusion depends on the stipulated independent sampling and the absence of a coupling mechanism or intermediate conditioning. A model can use shared randomness, richer joint outputs or repeated context updates. The example instead identifies a concrete failure mode of replacing a joint conditional distribution with a product of simultaneous marginals. Perfect tokenwise marginal estimates do not remove that failure. A report should therefore state how predictions at different positions are coupled during sampling, rather than inferring sequence-level fidelity from per-token accuracy alone.

6. Revealing context can restore dependence in the toy law

In the same two-token model, sample the first bit uniformly and then sample the second from the exact conditional distribution given the first. The second bit equals the first with probability one, so this two-stage process recovers the target law exactly. The improvement comes from conditioning, not from a general law that left-to-right decoding is superior.

A different reveal order works equally well in this symmetric example. A parallel algorithm that introduces an equivalent coupling can also recover the target. The lesson is to compare the actual information available at each update. If many positions are committed using an old context, later predictions within that batch cannot condition on those new commitments unless the implementation explicitly supports it. This creates a quality-compute tradeoff that depends on the data's dependence structure. Our finite law supplies an exact counterexample, while real text and image dependencies require empirical evaluation. One should not extrapolate its fifty-percent off-support rate to a trained diffusion model.

7. Iterative masked image generation is a policy

MaskGIT uses bidirectional context for masked image-token prediction and iteratively refines generation. The source demonstrates a useful generation paradigm distinct from a fixed raster-order autoregressive decoder. It does not make every confidence-based reveal or remasking schedule an exact sampler from a known joint distribution.

A reveal policy decides which predictions become context and which remain uncertain. A confident early mistake can influence later updates, while remasking can permit revision at the cost of additional computation and possible instability. Confidence is not automatically calibrated correctness, especially under a distribution or prompt shift. A fair experiment should record the masking schedule, sampling temperature, confidence definition and revision policy with the model checkpoint. The finite two-token construction can test whether a particular policy uses newly revealed context as intended, but it cannot predict image quality. For real visual tokens, reconstruction quality and the tokenizer's information bottleneck introduce additional limits before the sampling policy is considered.

Sources: [2]

8. An objective must be distinguished from a sampler

The masked diffusion language-model work of Sahoo and colleagues relates its training formulation to a mixture of masked language-model losses and studies efficient sampling strategies. This connection is useful because it makes explicit how denoising supervision is distributed across corruption levels. It should not be simplified into a claim that any masked-language objective and any reveal schedule define the same probabilistic model.

A likelihood bound, an estimator of that bound and a practical generation routine are different objects. The bound may rely on a specified forward process and reverse parameterization. The estimator may add time sampling or other approximations. The sampler may use fewer steps, cache results or adopt a heuristic reveal schedule. Evaluation should identify which object a reported number measures. Comparing a bound for one method with an exact log-likelihood for another requires careful labeling, even if both are expressed in similar units. The purpose is not to disqualify approximate metrics, but to prevent them from acquiring stronger semantics through ambiguous terminology.

Sources: [3]

9. Discrete score ratios change the reverse parameterization

Lou, Meng and Ermon introduce score entropy for discrete diffusion, focusing on ratios of probability values rather than a direct import of continuous score notation. This provides another route to parameterizing reverse dynamics. The inspected primary source supports the role of ratio estimation; its benchmark comparisons are not reproduced here.

A ratio can become numerically difficult when its denominator is small, and a learned estimate can be inaccurate in regions rarely seen during training. The practical sampler also has its own time discretization or transition approximation. These errors should be separated from the ideal reverse-process construction. In a finite model, exact probabilities make it possible to compute the true ratios and check normalization of resulting transitions. That is an appropriate unit test for a sampler implementation. In a large language model, the same exact reference is unavailable, so one needs diagnostics, controlled subproblems and empirical generation checks rather than assuming the learned ratio field is exact because the underlying formalism is principled.

Sources: [4]

10. Scale does not erase interface assumptions

The LLaDA paper studies language diffusion with forward masking and reverse token prediction at larger model scale, including pretraining and supervised fine-tuning. It is evidence that useful language capabilities can be studied outside a strictly autoregressive generation interface. Its reported task comparisons and case studies do not establish that one factorization or sampler dominates every other system under every budget.

For a deployed interface, one must specify sequence length, stopping behavior, prompt preservation, output revision and how streaming is exposed. A model that can revise earlier positions may provide useful global consistency but a different user experience from irrevocable left-to-right emission. A fixed-length generation window can require a policy for extending text or handling end markers. Those policies affect both quality and latency. Benchmarks should compare the actual service behavior needed by users, including time to first usable output and time to final stable output, rather than equating network-evaluation count with every aspect of responsiveness.

Sources: [5]

11. Continuous text diffusion has a different boundary

Diffusion-LM denoises continuous vector representations for controllable text generation. It is included as a contrast: diffusion used for text does not necessarily mean a categorical Markov chain over token identifiers. A continuous latent trajectory and the eventual mapping back to discrete text create a different modeling and evaluation boundary.

If a continuous vector lies between token representations, a rounding or decoding operation must choose a discrete output. Distances in that latent space need not correspond uniformly to semantic or syntactic changes. Conversely, categorical masking makes erasure explicit but introduces its own dependence on tokenization and corruption schedules. Comparing the approaches requires stating which variables are continuous, which are discrete and where information is discarded. A control method operating through latent gradients is not automatically available in the same form for a categorical model, and a categorical likelihood interpretation cannot simply be transferred to continuous embeddings. The interface should be described at the level of actual states and transitions.

Sources: [6]

12. Conditioning and editing require preservation rules

An infilling task can hold observed context fixed while generating missing positions. A practical sampler must decide whether fixed tokens are truly immutable, are repeatedly reinserted or are allowed to change under a soft condition. These choices define different tasks. A visually plausible completion is not sufficient evidence that a hard constraint was respected.

A test suite should include context tokens at every boundary, unusual mask patterns and incompatible requested constraints. It should verify unchanged fixed positions directly and report when no valid output is found. If a sampler uses rejection or repair, the extra work belongs in its budget. The two-token model offers a simple conditional test: fixing the first bit should determine the second under the stipulated target, while an inconsistent request for 01 should be identified as outside that target rather than counted as an ordinary successful completion. Real language does not have such a tiny exact support, but the principle of testing declared hard constraints independently of aesthetic quality still applies.

13. Vocabulary and length affect the compute contract

Discrete generation cost depends on sequence length, vocabulary representation, attention pattern and the number of network passes. A single pass over a long masked sequence can be much more expensive than a pass over a short one. Predicting all positions at each iteration can also repeat work for positions already considered settled, unless the implementation has a justified reuse strategy.

Tokenization changes the length and the meaning of a token-level metric. A lower per-token loss under a different vocabulary is not automatically a lower per-character or per-byte uncertainty. Likewise, comparing generation latency at equal token count can compare different amounts of visible text. Reports should include a stable content unit and the tokenizer version, alongside model parameters and pass counts. For images, spatial token count and decoder resolution introduce analogous issues. A useful benchmark fixes the input and output contract first, then measures quality and resources. It should not select whichever denominator makes a particular architecture appear most efficient.

14. Correlation tests complement aggregate text scores

The finite joint-law example can be extended with a correlation parameter c. Let equal pairs each receive probability one plus c divided by four, and unequal pairs each receive one minus c divided by four. Both marginals remain uniform. Independent simultaneous marginal sampling always produces the uniform joint law, whose total variation from the target is c divided by two. Exact conditional sampling matches the target for every c in the stated range.

This family separates marginal accuracy from dependence error continuously. It is not a model of natural-language semantics and should not be presented as a measured advantage of any architecture. Its value is diagnostic: an implementation claiming to sample the toy conditionals should reproduce the known joint probabilities as its sampling error decreases. A deterministic exact probability calculation can validate the expected result before a randomized simulation is added. For real models, analogous controlled tasks can reveal dependence and hard-constraint failures that a single aggregate perplexity or preference score may not explain.

15. What the executable illustrations verify

The first check constructs both corruption matrices, multiplies them over selected schedules and compares the result with direct enumeration of intermediate states. It verifies the absorbing-state rule, the product retention parameter and the chance-match term of uniform replacement. The plotted values use a fixed retention of 0.8 per step. They represent original-token observation probabilities under two different kernels, not a general information-theoretic measure.

The second check enumerates the four binary sequences, calculates marginals and joint probabilities, and compares independent and conditional generation. It confirms the half off-support mass at perfect correlation and the total-variation formula across several correlation values. Negative tests should swap matrix conventions, omit the chance-match term or treat simultaneous independent predictions as exact joint samples. These mistakes should fail explicit assertions. The checks validate the article's finite constructions only. They do not train D3PM, MaskGIT, MDLM, SEDD, LLaDA or Diffusion-LM, and they do not reproduce any cited quality or speed result.

16. The reverse model inherits the forward question

Discrete diffusion is not simply continuous diffusion with a different output layer. Its finite-state transitions define which information is erased or altered, while the reverse parameterization and reveal policy define how dependencies can be rebuilt. The selected sources illustrate several useful choices without collapsing them into one universal generation mechanism.

The practical conclusion is to publish a complete state-transition contract: tokenization, corruption, conditioning, reverse estimates, update coupling, stopping and resource accounting. Our finite examples show why a retention parameter is not always an original-token probability and why perfect marginals do not automatically produce a correct joint distribution. They do not rank real systems. Future work should combine exact small-state tests with realistic sequence evaluations, preserve hard-constraint failures and report stable-output latency alongside compute. That evidence can make parallel generation claims precise enough to compare and maintain without overstating what a mask, a loss or a sampling schedule guarantees.

Core equations and definitions

(1) Constructed absorbing mask kernel for ordinary token x and mask m.

(2) Uniform replacement over K ordinary symbols; equality includes accidental returns.

(3) Constructed two-token joint law with uniform marginals and c between zero and one.

(4) Exact finite-distribution result for independent uniform marginal sampling.

Constructed analytical figures

Comparison and reporting contract

Evaluation protocol

• Specify tokenizer, finite state space and mask semantics.
• Fix row/column transition conventions and test stochasticity.
• Verify forward composition against independent finite enumeration.
• Name reverse targets, likelihood bounds and sampling approximations separately.
• Record reveal/remask schedules and coupling between positions.
• Test hard context-preservation rules directly.
• Keep vocabulary and output-length units stable across comparisons.
• Measure full network passes, latency and stable-output behavior.
• Use exact correlated finite laws as diagnostic regressions.
• Report failures and bounds rather than inferring universal architecture rankings.

Limitations

• The finite kernels and binary laws are author-constructed models.
• Independent simultaneous sampling is an explicit toy assumption, not an intrinsic limitation of all parallel generators.
• No model training, benchmark perplexity or speed comparison is reproduced.
• Likelihood bounds and practical samplers may have different semantics.
• Continuous text diffusion is included as a contrast, not treated as categorical diffusion.
• The source selection is focused and not exhaustive.

Open research questions

• Which dependence diagnostics best predict failures of large parallel reveal batches?
• Can a sampler expose hard-constraint and stable-output contracts without sacrificing useful revision?
• How should tokenization-normalized likelihood and full-pipeline latency be compared across diffusion and autoregressive interfaces?

Evidence and its boundaries

D3PMs generalize categorical corruption beyond uniform transitions, including structured and absorbing kernels.

[1] — Choosing a kernel does not by itself establish reverse-model accuracy or a particular generation speed. Inspected: Primary abstract and version record on structured transition matrices, absorbing states and denoising objectives.

MaskGIT predicts masked image tokens with bidirectional context and iteratively refines generation.

[2] — Its schedule and confidence policy are algorithmic choices, not exact conditional sampling guarantees in all settings. Inspected: Primary abstract describing bidirectional masked-token prediction and iterative image refinement.

Masked diffusion training can be related to a mixture of masked-language-model losses with a specified objective.

[3] — A training objective or likelihood bound should not be confused with exact likelihood of every practical sampler. Inspected: Primary abstract and record concerning masked diffusion objectives and semi-autoregressive sampling.

SEDD uses a discrete score-entropy formulation to estimate ratios relevant to reverse dynamics.

[4] — Learned ratio estimates and a numerical sampler introduce errors separate from the ideal construction. Inspected: Primary abstract on score entropy and discrete distribution-ratio estimation.

LLaDA studies diffusion-based language modeling at larger scale using a masking and reverse-prediction formulation.

[5] — Reported comparisons and case studies do not establish universal superiority over autoregressive models. Inspected: Primary abstract and record on LLaDA's forward masking, reverse generation, pretraining and supervised fine-tuning.

Diffusion-LM uses continuous latent vectors for text generation, contrasting with direct categorical corruption.

[6] — Continuous embedding diffusion and discrete token diffusion have different state spaces and decoding boundaries. Inspected: Primary abstract describing continuous diffusion over word-vector sequences and controllable text generation.

References

1. Structured Denoising Diffusion Models in Discrete State-Spaces
   Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, Rianne van den Berg. NeurIPS 2021 (2021). https://arxiv.org/abs/2107.03006v3
   Source scope

   Primary abstract and version record on structured transition matrices, absorbing states and denoising objectives.

2. MaskGIT: Masked Generative Image Transformer
   Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, William T. Freeman. CVPR 2022 (2022). https://arxiv.org/abs/2202.04200
   Source scope

   Primary abstract describing bidirectional masked-token prediction and iterative image refinement.

3. Simple and Effective Masked Diffusion Language Models
   Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, Volodymyr Kuleshov. NeurIPS 2024 (2024). https://arxiv.org/abs/2406.07524v2
   Source scope

   Primary abstract and record concerning masked diffusion objectives and semi-autoregressive sampling.

4. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution
   Aaron Lou, Chenlin Meng, Stefano Ermon. ICML 2024 (2024). https://arxiv.org/abs/2310.16834v3
   Source scope

   Primary abstract on score entropy and discrete distribution-ratio estimation.

5. Large Language Diffusion Models
   Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, Chongxuan Li. arXiv preprint (2025). https://arxiv.org/abs/2502.09992v3
   Source scope

   Primary abstract and record on LLaDA's forward masking, reverse generation, pretraining and supervised fine-tuning.

6. Diffusion-LM Improves Controllable Text Generation
   Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, Tatsunori B. Hashimoto. NeurIPS 2022 (2022). https://arxiv.org/abs/2205.14217
   Source scope

   Primary abstract describing continuous diffusion over word-vector sequences and controllable text generation.