Before the Generator Learns: Visual Tokenization, Information Bottlenecks, and Sequence Cost

1. Tokenization creates the prior's world

A generative prior can learn only the distinctions represented in its training tokens. If two different inputs receive the same latent code sequence, a deterministic decoder using only that sequence cannot know which original input was encoded. It may produce a plausible representative, but plausibility and source reconstruction are different objectives. This information boundary exists before the choice of an autoregressive, masked or diffusion prior.

A useful evaluation therefore treats the tokenizer, prior and decoder as separate components with a shared contract. The tokenizer determines the latent alphabet, sequence layout and discarded information. The prior models a distribution over those sequences. The decoder maps sampled or encoded latents back into an output space. Errors at these stages can interact, but they should not be attributed indiscriminately to the final generator. We focus on the capacity and cost implications of that interface, using finite illustrations instead of extrapolating benchmark rankings across unrelated systems or domains.

2. Discrete representation and a learned prior are distinct

The VQ-VAE paper learns discrete encoder outputs and a prior over those representations. This architecture makes the separation between representing observations and generating latent samples explicit. The inspected primary source supports that design; it does not imply that a finite codebook preserves every property of the input or that its index count is an optimized compressed bitstream.

One can evaluate a tokenizer with the prior removed by encoding and decoding held-out inputs. One can then evaluate the prior's fit to those encoded sequences and the quality of decoded generated sequences. These experiments answer different questions. A strong reconstruction model can produce a latent distribution that is difficult for a fixed prior to learn. A simple latent distribution can arise because the tokenizer discarded meaningful variation. A prior can also generate code combinations that were rare or absent during tokenizer training, exposing decoder behavior outside its familiar latent region. The full pipeline needs all three forms of evidence rather than a single generation score.

Sources: [1]

3. Four inputs expose the bottleneck exactly

Consider a toy image source consisting of the four binary pairs 00, 01, 10 and 11, each equally likely. An encoder with only two possible code states cannot assign a distinct state to every input. For deterministic reconstruction, at most two of the four inputs can be recovered exactly by a decoder that sees only the code. The best exact-reconstruction probability is therefore one half.

This follows from a counting argument, not from an assumption about a neural-network optimizer. Each code state can decode to at most one exact source value, so M states can exactly represent at most M members of a uniform four-value source. Randomizing a decoder does not create additional information about which input was sent. The result does not prohibit plausible generation: a decoder can sample attractive outcomes. It limits exact recovery of a particular encoded input under the stated information constraint. For natural images, the relevant source distribution and distortion measure are more complex, but the distinction between reconstructing evidence and synthesizing plausible detail remains essential.

4. Distortion depends on what differences count

Use Hamming distance between the two-bit toy images, normalized by the two positions. With one reconstruction prototype, the minimum average distortion is one half. With two prototypes, it is one quarter. With all four prototypes, it is zero. These values can be verified by exhaustively enumerating prototype sets and assigning each input to a nearest prototype.

The calculation is deliberately different from exact-reconstruction probability. A representation can fail to reconstruct an input exactly while making only one small positional error. A perceptual measure may treat some changes as nearly irrelevant and others as important, while a scientific image may require precise preservation of a small feature. Choosing a distortion function is therefore a product and scientific decision, not an incidental scoring convenience. A report should state which information is permitted to change and which is a hard constraint. No single finite example determines that choice for real data, but it prevents a visually convincing decoder from being mistaken automatically for a faithful recorder.

5. Perceptual tokenization changes the reconstruction target

Esser, Rombach and Ommer combine a learned visual vocabulary with transformer-based composition, using a representation suited to high-resolution synthesis. The methodological lesson is that a tokenizer can be optimized for the needs of a generative pipeline rather than for literal pixel reconstruction alone. Its quality must be interpreted relative to those objectives.

A perceptual or adversarial objective can favor outputs that resemble the data distribution even when a particular fine detail differs from the source image. That may be desirable for creative generation and unacceptable for a task requiring evidentiary fidelity. Evaluation should therefore distinguish source reconstruction, distributional plausibility and downstream task preservation. A high-quality generated sample cannot establish that the tokenizer retained every source feature. Conversely, a small pixel error does not guarantee semantic equivalence. Useful test sets include text, repeated structures, small objects, unusual colors and domain-specific features whose importance may be poorly captured by a generic reconstruction metric.

Sources: [2]

6. Codebook size is a nominal capacity

For a sequence of L codes drawn from an alphabet of K values, the number of possible sequences is at most K raised to L. A fixed-width index representation needs L times the ceiling of log base two of K bits. These are capacity and storage-count statements. They do not assert that all codes are used, that sequences are independent or that a learned entropy model can compress them to that nominal count.

If a tokenizer uses only a small fraction of its codebook, nominal capacity can overstate the represented diversity. If codes are highly predictable from their context, an entropy coder may use fewer bits than fixed-width indices, but it also needs a probability model and a compatible decoding protocol. Side information and model weights add other costs. A tokenizer intended only as an internal generative representation need not be a deployable image codec. A paper should consequently avoid calling a raw index count a measured bitrate unless an actual bitstream, decoder and accounting boundary support that interpretation.

7. Scalar product codes change the quantizer design

Finite Scalar Quantization forms an implicit codebook from a small number of scalar dimensions, each quantized to a set of levels. This differs from directly learning a table of arbitrary vector prototypes. The source motivates a simpler discrete representation and evaluates it in selected tasks. Matching the product of level counts to a VQ codebook's size does not make the two models identical.

The product structure constrains how nearby continuous features map into discrete states. The encoder and decoder can adapt to that structure, while code utilization, gradient approximations and downstream prior complexity can change. A fair comparison should keep the latent sequence length and nominal capacity visible, then measure reconstruction and generation separately. It should also record quantization rules and any special handling at boundaries. An apparent improvement from a new quantizer can depend on a different encoder architecture or training objective. These are legitimate design choices, but isolating them prevents the codebook label from absorbing every change in the pipeline.

Sources: [3]

8. Spatial positions and residual depth are different axes

Residual quantization represents a feature using a stack of code components. Lee and colleagues use this structure with a corresponding generative prior. This can reduce the number of spatial positions while retaining additional code decisions at each position. Counting only the spatial grid can therefore understate the size or complexity of the latent description.

A specification should report spatial dimensions, residual depth, codebook sizes and the order or coupling used to predict the components. A prior may share computation across a stack, but that is an architectural property to measure rather than an automatic consequence of fewer positions. Reconstruction can improve when extra components refine a residual, while generation may become harder if the code stack has dependencies the prior models poorly. The evaluation should also consider truncation: if a system drops later residual components to save work or storage, the resulting quality and interface may differ from the full model. These distinctions matter when comparing a compact spatial representation with a genuinely shorter total token sequence.

Sources: [6]

9. One-dimensional sequences alter the layout contract

TiTok studies compact one-dimensional latent sequences instead of only fixed two-dimensional grids. Its source demonstrates that learned token layouts can be explored beyond a direct spatial downsampling interpretation. The reported compactness belongs to the trained model and evaluation setting, not to a lossless encoding guarantee for arbitrary images.

A one-dimensional latent position may represent information distributed across an image rather than a fixed local patch. That can be useful for compressing redundancy, but it changes how spatial editing or partial reconstruction should be interpreted. Replacing one latent token need not correspond to replacing one small region. A comparison should identify which spatial controls are native to the representation and which require additional conditioning or a separate model. Token order also affects an autoregressive prior's dependencies. The number of positions is therefore only one part of the interface; the meaning of positions and the decoder's response to changes deserve direct tests.

Sources: [5]

10. Tokenizer effects require controlled generator comparisons

The MAGVIT-v2 work emphasizes the importance of a shared visual tokenizer for image and video generation and reports results for its evaluated systems. Its title should be read in that experimental context, rather than promoted into a permanent universal ranking between language-model and diffusion approaches. Tokenizer quality can change the apparent strength of a downstream model family.

A useful controlled study holds the prior architecture and training budget fixed while changing the tokenizer, then separately studies whether each prior has been tuned fairly for the new representation. The first comparison isolates an interface intervention; the second estimates achievable system performance. Neither should be mislabeled as the other. For video, temporal reconstruction, motion consistency and token rate over time add further obligations. A shared vocabulary does not guarantee that every spatial or temporal detail is represented equally well. Evaluations should retain rare scenes and fast changes, not only average perceptual quality on familiar sequences.

Sources: [4]

11. Shorter sequences can reduce some quadratic work

For dense self-attention, the number of token pairs is proportional to L squared. Reducing a sequence from 256 positions to 64 reduces that pair count from 65,536 to 4,096, a factor of sixteen. If the alphabet has 1024 values, fixed-width nominal indices fall from 2560 bits to 640 bits, a factor of four. The two ratios differ because the quantities scale differently.

These are constructed accounting examples, not measured speedups or bitrates. Actual transformer cost includes projections, feed-forward layers, decoding, memory traffic and implementation choices. A shorter sequence may use wider hidden states, a more expensive tokenizer or more residual components. Autoregressive decoding with caching also has a different cost shape from full-sequence training. A complete experiment should report all changed dimensions and measure full-pipeline latency and memory. The simple pair-count curve is useful only when it remains labeled as a limited component of the computational model.

12. Reconstruction sets a floor for source-faithful tasks

If a tokenizer-decoder pair changes a medically relevant mark, a tiny label or a numerical plot before the prior is involved, a more accurate prior cannot reconstruct that particular lost observation from the same code alone. It may guess a plausible feature based on learned regularities, but guessing is not preserving source evidence. This distinction matters whenever generated or reconstructed outputs might be mistaken for measurements.

A task-specific evaluation should therefore begin with tokenizer-only reconstruction and independently check protected details. It should include counterfactual pairs that differ only in a small important feature, then test whether the encoder distinguishes them and the decoder preserves that distinction. The four-state toy model supplies the exact logic of this test, while real applications require carefully designed examples and expert criteria. The conclusion is not that every tokenizer must preserve every pixel. It is that discarded information must be compatible with the intended use, and any downstream synthesis should not conceal that information boundary.

13. Prior quality and latent coverage need separate tests

A latent prior can fit common code patterns while underrepresenting rare combinations. A tokenizer can also map unusual inputs into codes that the prior sees rarely, even when reconstruction is adequate. Generated samples and reconstructed samples consequently probe different distributions of latent inputs to the decoder.

We propose reporting tokenizer reconstruction on held-out inputs, prior likelihood or another clearly labeled fit measure on encoded sequences, and decoded generation quality under a fixed sampling policy. A latent coverage analysis should use stated bins or neighborhood definitions, because a generic code-frequency histogram does not capture dependencies among positions. Samples outside common latent regions deserve inspection for instability or unsupported detail. These diagnostics do not provide a complete measure of visual quality, but they localize failures. Without them, a poor final sample can be blamed on the prior when the tokenizer discarded the relevant distinction, or on the tokenizer when the prior generated an implausible code combination.

14. Tokenizer updates are model-interface updates

Changing a codebook, quantizer, encoder or latent scaling can alter the meaning of every token consumed by a prior. A prior trained on one tokenizer cannot be assumed compatible with another merely because both use integer arrays of the same shape. A code index is a symbol in a learned interface, not a stable semantic identifier across independently trained models.

Artifact versioning should bind the tokenizer and decoder to the prior checkpoint. Conversion procedures need their own validation, including reconstruction and generation checks on a fixed reference set. A service that exposes editing operations should also recheck the spatial or temporal meaning of those controls after a tokenizer update. The same principle applies to entropy-coded export: the probability model and index ordering must agree between encoder and decoder. Successful tensor loading or successful parsing of an index file establishes format compatibility, not semantic equivalence of the reconstructed output.

15. The finite checks verify capacity and accounting

The first construction enumerates the four binary source images and every prototype set of one, two or four states. It assigns each source to a nearest prototype, calculates normalized Hamming distortion and counts exact reconstructions. The optimal distortions are one half, one quarter and zero; the maximum exact-reconstruction probabilities are one quarter, one half and one. These values should be computed by enumeration rather than hard-coded into the checking script.

The second construction fixes K at 1024 and varies L over a declared list. It computes fixed-width index counts and dense-attention pair counts, preserving their different units. Negative tests should confuse code capacity with actual entropy, omit residual depth or describe pair-count reduction as measured latency; the metadata checker should reject those labels. No trained tokenizer, image codec or generative prior is evaluated by these calculations. Their purpose is to make the article's limited mathematical claims reproducible while leaving empirical visual quality to an appropriately designed experiment.

16. A useful token is a maintained information contract

The selected sources explore different ways to create visual latent interfaces: learned vector codes, perceptual vocabularies, scalar product quantization, shared image/video tokens, compact one-dimensional sequences and residual stacks. Their results show why the representation can strongly affect a generative system, but no single token count summarizes every useful property.

A credible comparison states what information survives, how the prior models it, how the decoder reconstructs it and which computational components are actually reduced. Our finite examples separate exact recovery, distortion, nominal capacity and pair-count cost. They do not rank real tokenizers. Future evaluations should preserve rare details, bind checkpoint versions across components and report reconstruction and generation as distinct tests. This makes compact representations easier to use responsibly: a reader can see which savings are mathematical counts, which are measured engineering improvements and which assumptions define the information the system is allowed to discard.

Core equations and definitions

(1) Nominal code capacity and fixed-width index bits. Excludes entropy coding, side information and model weights.

(2) Uniform four-value toy source with at most M decoder code states; exact reconstruction, not perceptual quality.

(3) Constructed normalized Hamming distortion for a selected prototype set C.

(4) Dense-attention pair-count ratio, not a measured full-model speedup.

Constructed analytical figures

Comparison and reporting contract

Evaluation protocol

• Freeze tokenizer, decoder, prior and latent scaling versions.
• Specify codebook sizes, sequence layout and residual depth.
• Measure tokenizer-only reconstruction before generation.
• Test protected fine details with controlled input pairs.
• Evaluate prior fit and decoded samples separately.
• Report nominal capacity separately from actual coded bitstreams.
• Keep content units stable across resolution and tokenizer comparisons.
• Measure full-pipeline latency and memory, not only attention pair counts.
• Inspect rare and unfamiliar latent combinations.
• Revalidate semantic compatibility after tokenizer or decoder updates.

Limitations

• Four binary images are an explanatory finite source, not a natural-image distribution.
• Hamming distortion does not model human visual quality.
• Nominal index counts are not measured image compression rates.
• Dense-attention pair counts omit other network and system costs.
• No published reconstruction or generation metric is reproduced.
• The primary-source selection is focused rather than exhaustive.

Open research questions

• Which controlled detail-preservation tests predict downstream failures better than aggregate reconstruction scores?
• How should latent coverage and prior difficulty be compared across incompatible tokenizer layouts?
• Can tokenizer version metadata make editing and entropy-decoding compatibility independently checkable?

Evidence and its boundaries

VQ-VAE learns discrete latent representations that can be modeled by a separate prior.

[1] — Discrete codes alone do not ensure optimal rate-distortion behavior or preservation of every detail. Inspected: Primary abstract and record on VQ-VAE's discrete encoder outputs and learned prior.

A learned image representation can combine local inductive structure with a transformer prior over visual tokens.

[2] — Perceptual or adversarial reconstruction objectives are not guarantees of pixel-faithful recovery. Inspected: Primary abstract on learned visual vocabularies and transformer composition.

FSQ forms an implicit codebook from products of fixed scalar quantization levels.

[3] — Matching nominal codebook size does not make every quantizer's learned geometry or downstream quality identical. Inspected: Primary abstract describing low-dimensional scalar levels and the implicit product codebook.

Visual tokenizer design can materially affect downstream image and video generation results.

[4] — The paper's title is a claim for its evaluated systems, not a universal ranking of language and diffusion models. Inspected: Primary abstract and record on MAGVIT-v2's image/video vocabulary and reported controlled comparisons.

TiTok studies image tokenization through compact one-dimensional latent sequences rather than only fixed two-dimensional grids.

[5] — A demonstrated token length is not a lossless encoding guarantee for arbitrary images or domains. Inspected: Primary abstract on TiTok's one-dimensional compact latent sequences.

Residual quantization uses multiple code components to represent a feature while changing the prior's prediction structure.

[6] — Fewer spatial positions need not mean fewer total code decisions or an equivalent entropy-coded rate. Inspected: Primary abstract on stacked residual codes and a prior predicting those code stacks.

References

1. Neural Discrete Representation Learning
   Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu. NeurIPS 2017 (2017). https://arxiv.org/abs/1711.00937v2
   Source scope

   Primary abstract and record on VQ-VAE's discrete encoder outputs and learned prior.

2. Taming Transformers for High-Resolution Image Synthesis
   Patrick Esser, Robin Rombach, Björn Ommer. CVPR 2021 (2021). https://arxiv.org/abs/2012.09841v3
   Source scope

   Primary abstract on learned visual vocabularies and transformer composition.

3. Finite Scalar Quantization: VQ-VAE Made Simple
   Fabian Mentzer, David Minnen, Eirikur Agustsson, Michael Tschannen. arXiv preprint; ICLR 2024 paper (2023). https://arxiv.org/abs/2309.15505v2
   Source scope

   Primary abstract describing low-dimensional scalar levels and the implicit product codebook.

4. Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
   Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, Lu Jiang. ICLR 2024 (2024). https://arxiv.org/abs/2310.05737v3
   Source scope

   Primary abstract and record on MAGVIT-v2's image/video vocabulary and reported controlled comparisons.

5. An Image is Worth 32 Tokens for Reconstruction and Generation
   Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, Liang-Chieh Chen. NeurIPS 2024 (2024). https://arxiv.org/abs/2406.07550
   Source scope

   Primary abstract on TiTok's one-dimensional compact latent sequences.

6. Autoregressive Image Generation using Residual Quantization
   Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, Wook-Shin Han. CVPR 2022 (2022). https://arxiv.org/abs/2203.01941v2
   Source scope

   Primary abstract on stacked residual codes and a prior predicting those code stacks.