Tokens Are Not Units of Meaning: Probability Accounting Across Segmentations

1. A token is an engineering unit

A token can represent a character fragment, a word, a byte sequence or a dynamically selected patch. Its boundaries are determined by an encoding procedure rather than by a universal unit of meaning. This matters whenever token counts are used to compare training budgets, context windows, likelihoods or serving costs across models.

A tokenizer can reduce sequence length while changing the distribution of symbols the model must predict. It can make one language or notation compact and another fragmented. It can normalize text in a way that removes distinctions important to a task. These changes are not captured by vocabulary size alone. This review focuses on probability accounting and evaluation units. An exact finite example shows how several token sequences can decode to one string, so canonical-segmentation likelihood need not equal string likelihood. A second construction shows why token perplexity can differ solely because the unit changes. The examples are mathematical controls, not empirical comparisons of deployed models. They establish which quantities must be aligned before a claim about tokenization efficiency or language-model quality is interpretable.

2. What the selected source records contribute

Sennrich and colleagues study subword units for rare-word translation. Kudo studies probabilistic alternative segmentations as subword regularization. SentencePiece provides a tokenizer and detokenizer designed to operate from raw text. These works motivate treating segmentation as a modeled design choice rather than an invisible preprocessing constant.

ByT5 studies byte-level sequence modeling, CANINE studies character-level encoding without a fixed subword tokenizer, and Byte Latent Transformer studies dynamically sized byte patches. These approaches use different computational units and therefore require careful budget and metric comparisons. Our inspection covers their original abstracts and bibliographic records. We do not reproduce source benchmarks or infer that one representation is best for every language, modality or workload. The original constructions below concern finite probability distributions and reversible decoding assumptions. They make no claim about the quality of a trained byte, character or subword model. Their purpose is to expose accounting errors that can otherwise make an apparently precise performance comparison depend on incompatible units.

Sources: [1], [2], [3], [4], [5], [6]

3. Encoding and decoding are different maps

Let E map a string to a canonical token sequence, and let D map a token sequence back to a string. If D(E(s)) equals s, the canonical encoding round-trips. That does not imply that E(D(t)) equals t for every token sequence t. Several token sequences can decode to the same string, especially when the vocabulary contains both a whole fragment and its smaller parts.

This distinction matters for generative models because they assign probabilities to token sequences, not directly to canonical encodings alone. If decoding is deterministic and the token distribution is normalized over terminated sequences, the probability of a string is the sum of probabilities of all token sequences that decode to it. The probability of one canonical sequence is only one term in that sum. Some systems constrain generation to canonical tokenizations or use normalization that changes the string space; those choices must be stated. A round-trip test on encoded inputs is necessary but not sufficient to characterize the distribution induced by unrestricted generation. The finite example below makes the difference explicit without relying on implementation details of a particular tokenizer.

4. An exact segmentation example

Consider a finite generator whose possible terminated token sequences include a single token AB and the two-token sequence A,B. Both decode to the string ab. Assign probabilities 0.3 and 0.2 to these sequences, respectively, with the remaining probability assigned to other strings. The decoded-string probability of ab is 0.5, while the canonical probability is 0.3 if the encoder selects AB.

The negative log probability of the canonical sequence is therefore larger than that of the decoded string by log base two of 0.5 divided by 0.3, measured in bits. The example assumes a complete finite distribution with explicit termination, so no unaccounted continuation probability is hidden. Figure one varies the mass assigned to the alternative segmentation while retaining the canonical mass and a valid remaining distribution. It shows the increasing gap between canonical code length and string code length. This is not a criticism of using canonical likelihood as a training objective. It is a distinction between two valid but different quantities. A report should name which one it measures and avoid presenting them as interchangeable.

5. Marginalization can be computationally difficult

In the finite example, all segmentations can be enumerated. In a large autoregressive model, alternative tokenizations can have different histories and therefore different conditional probabilities. Summing over them may require substantial computation. A simple dynamic program is available only when the model's dependence structure permits the needed factorization.

This limitation does not justify silently equating canonical likelihood with string likelihood. It justifies reporting canonical likelihood as a tractable metric with a defined scope. Approximate marginalization methods should disclose their proposal distribution, coverage and uncertainty. If they sample only some segmentations, their estimate may miss substantial probability mass. Conversely, a tokenizer with unique decoding paths under its generation constraints may avoid the ambiguity, but that property must be established for the actual pipeline. The broader lesson is that probability accounting depends on the combination of vocabulary, decoder and sequence model. A tokenizer specification alone is insufficient. The model's treatment of end markers, special tokens and invalid byte sequences also determines which outcomes contribute to the probability of a decoded string.

6. Perplexity depends on the denominator

For a sequence assigned total negative log probability L nats and represented by N tokens, token perplexity is exp(L/N). If the same information is divided into more units while the total code length stays fixed, the reported per-unit perplexity decreases. That change does not necessarily indicate a better predictive distribution over strings.

Figure two uses a declared fixed total code length for a fixed byte string and varies the number of tokens used to represent it. The token perplexity changes while bits per byte remains constant. This construction deliberately holds total probability fixed to isolate the unit effect. Real tokenizers also change the learned model and its probability assignment, so their differences are more complex. Nevertheless, the control shows why token perplexities from incompatible vocabularies cannot be ranked naively. A fair comparison can use a shared underlying unit such as bytes or characters, provided normalization and encoding are aligned. Even then, the metric describes predictive code length, not semantic correctness or task utility. Those outcomes require separate evaluation.

7. Normalization changes the sample space

A preprocessing pipeline may normalize Unicode, whitespace, case or punctuation before tokenization. If distinct original strings collapse to the same normalized string, the model no longer represents their distinction through that channel. This may be desirable for a task, but it changes what a likelihood or round-trip claim means.

A benchmark should preserve the original text and record the exact normalization procedure. Byte-normalized metrics are comparable only if the bytes being counted correspond to the same text representation. Counting original UTF-8 bytes while scoring a normalized string can create a mismatch. The same problem arises when invalid sequences are replaced or when line endings are changed. A model may appear robust because the preprocessing removes variation before the neural computation. That is a valid pipeline property, but it should not be attributed solely to the model. For scientific and code documents, normalization can alter identifiers, mathematical symbols or formatting with functional meaning. Evaluation should include these cases rather than assuming that all textual differences are semantically irrelevant.

8. Multilingual efficiency requires a distribution

A tokenizer's average tokens per character depends on the language mix, domain, script and normalization of the evaluation corpus. A single global average can hide severe fragmentation in a minority language or specialized notation. Vocabulary size does not reveal how representational capacity is allocated across those distributions.

A useful study reports stratified statistics and defines the sampling weights. It should distinguish bytes, Unicode code points and grapheme clusters, which are different units. It should also separate sequence-length efficiency from model accuracy and from serving cost. A compact tokenization can reduce attention work but may require a larger output vocabulary or change statistical sharing. A fragmented tokenization may increase the number of prediction steps while preserving useful compositional structure. Neither direction is universally superior. The relevant question is performance under a specified workload and resource budget. Historical source experiments motivate these tradeoffs, but the present review does not measure them. Its accounting framework prevents a token-count advantage on one corpus from being promoted into a universal claim about language fairness or intelligence.

9. Context budgets are representation-dependent

A fixed token limit corresponds to different amounts of text under different tokenizers. A system with a larger advertised token window may expose less actual content for a particular language or document type. The comparison becomes especially misleading when one method uses bytes and another uses subwords.

For document tasks, report the amount of original text available to the model, the truncation policy and the location of omitted evidence. If a tokenizer change alters which parts of a document fit, a downstream accuracy change may reflect information availability rather than improved reasoning. A controlled experiment can hold the original text fixed and vary computational budget, then separately hold token budget fixed to measure the serving constraint. Both views are useful, but they answer different questions. The same issue affects batching and padding. A tokenizer that reduces average length may also reduce variance, improving hardware utilization. These operational benefits should be measured directly rather than inferred from vocabulary statistics. The unit of the context window is part of the model interface and must be translated into the user's actual workload.

10. Training budgets need multiple denominators

Equal numbers of training tokens do not necessarily imply equal exposure to text or equal computation. A byte model and a subword model can process different amounts of original content under the same token count. They can also have different vocabulary projection costs, sequence lengths and architectural layouts.

A fair comparison should report original bytes or documents, model parameters, training FLOPs under a stated accounting rule, and wall-clock resources when relevant. No single denominator resolves every tradeoff. Matching text exposure may leave compute unequal; matching compute may leave exposure unequal. The report should make those choices explicit and, where possible, provide both comparisons. It should also account for tokenizer training data, which can introduce information unavailable to another method. A tokenizer learned from the evaluation corpus creates a different experimental condition from one fixed beforehand. The finite perplexity control does not prescribe the correct budget for every study. It shows why a token count cannot serve as a universal proxy for information, data volume and computation simultaneously.

11. Special tokens can carry hidden assumptions

Beginning, end, padding, separator and control tokens influence both the probability model and the interface. A likelihood calculation that omits termination probability measures a prefix event rather than a complete-string event. A decoder that strips special tokens can map additional token sequences to the same visible text.

These details should be included in the probability-accounting contract. For example, two sequences differing only in a stripped control token may contribute to the same displayed output but have different downstream effects before rendering. A text-only evaluation can miss that distinction. Padding masks also matter: counting padded positions in a denominator can distort average loss. A robust implementation test checks complete sequences, termination, invalid tokens and round trips for edge cases. It should compare the scored representation with the displayed representation. The goal is not to overcomplicate every model report, but to make the basic units unambiguous. Without that clarity, apparently small preprocessing choices can change the meaning of the reported probability or cost.

12. Robustness is task-specific

Byte and character representations can avoid certain out-of-vocabulary failures, while subword representations can exploit reusable fragments. Neither property alone establishes robustness to misspellings, morphology, adversarial formatting or code syntax. The effect depends on the model, training data and task.

A useful test suite includes controlled perturbations with preserved meaning and perturbations that genuinely change meaning. A model should not be rewarded for invariance when a changed identifier or numerical symbol should alter the answer. Conversely, harmless formatting variation should not cause unjustified failure. The evaluation should report the perturbation process and distinguish recognition from generation. A model may understand a noisy input but produce malformed output under the same representation. These considerations connect tokenization to end-to-end behavior rather than treating it as a standalone compression contest. The source records offer examples of alternative representations; the present analysis does not reproduce their robustness results. It specifies the evidence required before a new tokenizer is described as more robust for a particular application.

13. A reproducible comparison protocol

Begin with a versioned corpus preserving raw bytes and normalization metadata. Evaluate round-trip behavior, sequence length distributions and special-token handling before training. Define the probability event being scored: canonical token sequence, prefix or fully terminated decoded string. Use a shared underlying denominator when comparing code lengths across vocabularies.

Then conduct matched-exposure and matched-compute comparisons where feasible. Report downstream task quality, latency, memory and truncation effects separately. Stratify by language, domain and document length, retaining rare but consequential cases such as mathematical notation and source code. If approximate string marginalization is used, report its coverage and numerical error checks. Keep tokenizer training data separate from final evaluation data. The protocol is prospective; this article does not train competing language models. Its finite constructions provide sanity checks that a metric implementation should pass before producing comparative claims. A study that fails those elementary checks may still generate attractive charts, but their numerical precision would not repair the underlying mismatch in what was measured.

14. What the executable controls establish

The companion program defines a normalized finite distribution over terminated token sequences, decodes each sequence and sums masses by resulting string. It verifies that the canonical probability is no greater than the corresponding string probability and that total decoded probability remains one. It checks the code-length gap formula for several alternative-segmentation masses.

The second control computes token perplexity and bits per byte from a fixed total code length, verifying that the latter stays constant while the former changes with token count. Boundary checks include zero alternative mass and complete accounting of termination. These calculations validate the two figures and their interpretation. They do not estimate the segmentation ambiguity of a real model or prove that any tokenizer achieves a particular compression rate. The data are constructed and labeled accordingly. A reader can reproduce them with the standard library and inspect every assumption. This is arithmetic verification of an accounting distinction, not a new benchmark result.

15. Reading a tokenizer claim critically

An informative claim identifies the corpus, unit and downstream objective. Saying that a tokenizer uses fewer tokens is meaningful only after specifying what text and normalization were measured. Saying that a model has lower perplexity is meaningful only after specifying the probability event and denominator. Saying that a representation is more efficient requires a workload and resource measure.

These qualifications do not diminish genuine improvements. They make improvements portable and testable. A method can be better for a particular language mix or document type without being universally superior. It can improve serving latency while leaving semantic quality unchanged, or improve robustness while increasing sequence length. A strong report exposes these tradeoffs rather than compressing them into a single ranking. The finite examples show how misleading comparisons can arise even when every individual arithmetic operation is correct. The problem is the unit and event being compared. Aligning them is a prerequisite for interpreting the result, not an optional refinement after the benchmark has been run.

16. Open questions and limits

Future work can make tokenizer evaluation more transparent by publishing raw-text-aligned likelihoods, versioned normalization rules and uncertainty estimates for approximate marginalization. It can study how representation choices interact with multilingual data imbalance and with tasks requiring exact symbolic output. Dynamic patching raises additional questions about how adaptive computational units should be compared with fixed tokens.

This review does not settle those questions. It supplies a finite probability example and a unit-normalization control, supported by a focused primary-source overview. Its central conclusion is that tokens are model-specific accounting units, not universal semantic atoms. A credible comparison must connect them back to the text, information event and workload of interest. Once those links are explicit, differences in sequence length, likelihood and cost can be discussed without confusing one for another. That is a modest but necessary foundation for evaluating both conventional subword models and newer byte- or character-based alternatives.

Core equations and definitions

Constructed analytical figures

Comparison and reporting contract

Evaluation protocol

• Version raw text and normalization.
• Specify canonical, prefix or complete-string likelihood.
• Use shared underlying denominators across vocabularies.
• Report matched exposure and matched compute separately.

Limitations

• No language model or tokenizer benchmark is run.
• Finite distribution has explicitly enumerated terminated outcomes.
• Real string marginalization may be computationally difficult.

Open research questions

• How can approximate string likelihood be audited efficiently?
• Which workload units best compare adaptive byte patches and subwords?
• How do normalization choices affect exact scientific and code tasks?

Evidence and its boundaries

Studies subword units for rare-word translation.

[1] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Studies probabilistic alternative segmentation as regularization.

[2] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Provides raw-text subword tokenization and detokenization.

[3] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Studies byte-level sequence models and compute tradeoffs.

[4] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Studies character-level encoding without a fixed subword vocabulary.

[5] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Studies dynamic byte patches as computational units.

[6] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.