Rank Is Not a Capacity Certificate: Attention Bottlenecks Beyond the Score Matrix

1. Four objects hidden behind one word

An attention head transforms a sequence through several operations, and the word rank is frequently attached to all of them as if it described a single capacity. The score matrix compares queries with keys. A mask can remove comparisons. Row normalization turns the permitted scores into a probability distribution. Multiplication by values produces features, which may then be projected and added to a residual stream. Each stage changes the question that rank answers.

This article examines those distinctions using an exact two-token construction and a general output-error inequality. The objective is not to deny bottlenecks. It is to identify the mathematical object to which a claimed bottleneck applies, and therefore which experiment could support or refute it. A low-dimensional score parameterization constrains attainable log-odds, but nonlinear normalization can increase ordinary matrix rank. Conversely, a full-rank probability matrix can be numerically close to a rank-one matrix or can multiply values that erase its distinctions. A trained system may succeed through other heads or residual pathways. Statements about model capacity therefore need a specified input domain, target function, parameter budget, precision and approximation tolerance.

2. What the selected primary records establish

Vaswani and colleagues introduce the Transformer architecture built around attention mechanisms. Bhojanapalli and colleagues identify a head-dimension bottleneck and analyze an alternative parameterization. Dong and colleagues study degeneration in pure attention networks, explicitly distinguishing architectures with skip connections and multilayer perceptrons. These are different architectural questions and should not be collapsed into one claim about every attention matrix.

Yun and colleagues establish approximation results for specified classes of sequence mappings, with positional information changing the symmetry restriction. Choromanski and colleagues use random features to approximate attention kernels in Performers. Katharopoulos and colleagues formulate linear attention with recurrent evaluation. These latter approaches motivate separating exact representation from approximation and computational organization. The original records and abstracts support these broad distinctions; their complete proofs and benchmark pipelines are not independently audited here. We do not transfer a historical performance ranking to current models or infer that an approximation theorem guarantees a useful finite-budget training outcome. The rest of the article develops declared examples whose algebra can be checked independently of those papers. They are interpretive controls for language used in capacity discussions, not replacements for the source theorems.

Sources: [1], [2], [3], [4], [5], [6]

3. Scores constrain log-odds, not normalized rank alone

Let Q and K each have n rows and d columns. Before masking, S=QK transpose has rank at most d. Row-wise softmax applies exponentiation and a row-specific normalizer. This map is nonlinear, so the ordinary rank inequality for a product cannot simply be carried through it. A useful invariant is instead the difference of log probabilities within a row. For strictly positive attention, log Aij minus log Aik equals Sij minus Sik.

Consequently, a low-rank score family restricts a matrix of log-odds relative to a reference column. That restriction concerns which probability patterns can be represented exactly. It does not require every represented probability matrix to have low ordinary rank. It also depends on positivity: a masked-out probability is zero and has no finite logarithm. Analyses using log-odds must either restrict to the allowed support or handle masks separately. An empirical rank plot of A alone therefore does not settle the expressive question posed by a low-dimensional Q and K. A better diagnostic compares target log-odds with the feasible score family, then measures the resulting task-level error after values and residual composition.

4. An exact rank-one score construction

Take two queries q=(1,-1) and two keys k=(1,-1), both with one feature. Introduce a nonnegative scalar a multiplying their outer product. The score matrix has rows (a,-a) and (-a,a). For a greater than zero it has rank one. Its normalized matrix has diagonal entries p=1/(1+exp(-2a)) and off-diagonal entries 1-p. The two eigenvectors are the uniform direction and the contrast direction.

The corresponding eigenvalues are one and 2p-1=tanh(a). Thus for every finite a greater than zero the attention matrix is full rank, although the score matrix is rank one. At a=0 both rows are identical and the attention rank drops to one. This example is not an assertion that a single scalar head can realize any two-by-two stochastic matrix. It realizes a restricted symmetric family. That qualification is essential: exhibiting one full-rank matrix refutes an incorrect universal rank bound but does not establish universal expressivity. Figure one plots the exact contrast eigenvalue and the determinant, which coincide for this family. The executable check verifies both against the explicit softmax calculation.

5. Numerical rank changes the practical interpretation

Exact rank treats an arbitrarily small nonzero singular value as fully informative. A finite-precision or noisy computation cannot necessarily use that distinction. In the symmetric example, the singular values are one and tanh(a). As a approaches zero, the contrast singular value approaches zero and the two rows become nearly identical. A thresholded numerical rank would then report effective collapse even though exact algebra still reports rank two.

A diagnostic should state its threshold and whether it is absolute or relative to the leading singular value. It should also state whether the matrix was formed in training precision or recomputed at higher precision. Otherwise differences attributed to architecture may be consequences of the diagnostic itself. For this example, the condition number is one divided by tanh(a), which becomes large near zero. However, condition number is relevant to inversion or recovering distinctions, not automatically to every downstream use. If the desired task only needs the uniform component, losing the contrast may be harmless. A task-dependent analysis asks whether the target variation lies in the attenuated direction rather than treating every small singular value as a failure.

6. Saturation creates a second tradeoff

Increasing a makes the contrast direction stronger, but it also changes sensitivity to the score scale. The derivative of tanh(a) is one minus tanh(a) squared. It approaches zero as a grows. The forward map then preserves the two-token contrast well while becoming less responsive to incremental changes of the score magnitude. Figure two displays contrast transmission and its derivative on the same dimensionless scale, with separate labels.

This is a local property of the constructed scalar parameter, not a theorem that high attention concentration always harms training. Gradients with respect to queries, keys, values and earlier representations have additional factors and pathways. Even within this example, a different parameterization changes the derivative with respect to its parameter. The useful lesson is more limited: an attention matrix can look well-conditioned as an operator on values while the normalization becomes saturated as a map from scores. Reporting only one of these measurements conflates forward information transmission with trainability. A careful study varies score scale while controlling value magnitude and records both the functional response and gradients under the actual parameterization.

7. Values can remove every apparent distinction

Set both value rows equal to the same vector v. Any row-stochastic attention matrix then returns v at every output position. Its rank can be full, its rows can be sharply different, and its score parameterization can be expressive, yet the values carry no position-specific distinction to recover. More generally, the rank of AV is bounded by the smaller rank of A and V. That elementary product bound applies after softmax because the final operation is linear in V.

The converse also deserves care. A low-rank output on one observed input does not establish that the architecture cannot represent high-rank outputs on other inputs. It may reflect redundancy in that example or a deliberately invariant solution. Measuring values, attention and output together helps localize the bottleneck. For a classifier using only a pooled output, a token-level rank metric can be poorly aligned with the task. The experiment should therefore include a target requiring a known contrast and a control where that contrast is irrelevant. Without these paired tasks, a change in rank can be visually dramatic while providing little evidence about functional capacity.

8. Masks introduce support and topology

A causal mask sets future comparisons to zero after normalization. If every permitted diagonal score is finite, the resulting lower-triangular attention matrix has a positive diagonal and therefore full rank. This algebraic observation holds even when the unmasked score matrix has a low-dimensional factorization. It is another reason that a rank claim must state whether masks are included.

Full rank here does not imply that arbitrary historical information is recoverable under finite precision. Diagonal entries may be tiny, and repeated transformations can attenuate relevant directions. Nor does triangular invertibility permit a causal model to access future tokens. Algebraic inversion of an already formed matrix is not the same operation as online generation with restricted information. A useful evaluation keeps support constraints separate from numerical conditioning and from task directionality. It can compare identical scores under causal and unrestricted masks, but it must acknowledge that this changes the computational problem. Treating a mask as a negligible implementation detail can produce misleading interpretations of both matrix rank and sequence dependence, especially when sequence lengths vary.

9. Multiple heads are not independent certificates

Concatenating heads introduces several value channels and several comparison patterns, but counting heads alone does not count independent functions. Heads can converge to redundant patterns, share an input subspace, or be combined by an output projection that discards their differences. Conversely, a head that looks redundant under an attention-only similarity metric may carry different values and contribute a distinct function.

The proper unit of comparison is therefore the complete head output and its role in the residual stream. An ablation that removes a head changes both representation and normalization of the surrounding network; retraining and inference-only removal answer different questions. Parameter-matched comparisons must account for head width, total embedding dimension, projection matrices and sequence length. The score rank bound depends on each head's dimension, while the final network's available computation depends on their composition. Our two-token construction cannot predict how these tradeoffs resolve on a large benchmark. It supplies a compact negative control: any measurement system claiming that every width-one head necessarily produces rank-one normalized attention must fail on this exact example.

10. Residual paths change the object being studied

If an attention block outputs X+AV after compatible projection, a rank-deficient attention branch does not force the total output to be rank deficient. The identity contribution may preserve distinctions that the attention branch averages away. A multilayer perceptron adds another nonlinear pathway. Removing these components to obtain a clean theorem creates a useful restricted model, but its conclusions must not be silently assigned to the full architecture.

The distinction also affects experimental controls. Comparing a pure attention stack with a residual Transformer changes more than depth. A decline in token diversity may be expected in the former and counteracted in the latter. The appropriate question is whether the proposed mechanism persists after restoring the components present in the deployed model. A residual path can preserve information without making that information accessible to the final readout, so preservation and utilization remain separate. One can test both by injecting a known contrast, tracking its norm through the residual stream, and measuring whether the task head responds to it. These measurements locate information loss more precisely than a single attention-spectrum plot.

11. Approximation error must reach the output

Suppose an approximation produces Ahat and the exact attention is A. For each output row, the Euclidean norm of the difference after multiplication by V is bounded by the row's L1 attention error times the largest Euclidean norm of a value vector. This follows by applying the triangle inequality to the weighted sum of value vectors. It is a deterministic bound requiring no distributional assumptions.

The inequality explains why a small matrix approximation error can be insufficient when values are large, and why a larger attention error can be harmless when it lies along value redundancies. It also exposes the norm convention: an entrywise maximum bound would acquire a factor involving sequence length, whereas the row-L1 form already sums over keys. A study of efficient attention should report the approximation target, norm and value scale, then measure output and task errors. Runtime savings alone do not establish equivalent behavior. Conversely, failure to reproduce every entry of dense attention does not prove failure on the intended task. The relevant loss is attached to the actual model output under the stated input distribution.

12. A discriminating evaluation design

Begin with three controlled families: the two-token contrast, identical values with varying attention, and causal masks with fixed scores. These controls test whether the measurement pipeline distinguishes score rank, attention rank and output rank. Next vary head width, number of heads and score scale while holding total parameter accounting explicit. Record exact and thresholded singular spectra, output error, gradient norms and task performance.

For learned models, use multiple independent training seeds and retain failed runs. Analyze length generalization separately from in-distribution accuracy because a fixed width can interact with a changing number of tokens. Keep preprocessing, precision and mask semantics stable across candidates. A useful target family should include both low-dimensional summaries and tasks requiring many independent associations. Otherwise the benchmark may reward compression while the paper interprets it as evidence of general memory capacity. Report uncertainty over tasks and seeds rather than using individual tokens as independent experimental replicates. No such neural training campaign is executed in this review; this paragraph specifies the experiment needed to move from our algebraic controls to an empirical architectural conclusion.

13. What the executable checks establish

The accompanying standard-library program computes stable softmax for the two-by-two score family, verifies row normalization, checks its determinant against tanh(a), and confirms exact collapse at zero. It checks the forward contrast response using opposite value vectors and verifies that identical values erase the distinction. A finite set of perturbations also tests the stated output bound with independently computed row-L1 errors.

These checks validate arithmetic, figure data and the declared finite constructions. They do not prove a theorem about all learned attention heads, recover a source paper's training results, or characterize numerical behavior on every accelerator. The program uses ordinary floating-point arithmetic, reports explicit tolerances, and avoids inversion near zero. Its near-zero points are interpreted through conditioning rather than unstable numerical rank estimates. All plotted points are generated from the same formulas used by the checks, so a reader can inspect the link between a caption and its data. This is reproducibility of an analytical illustration, not replication of a neural experiment. Keeping that scope explicit prevents a small successful script from being promoted into evidence for a much broader capacity claim.

14. A better language for bottlenecks

An informative capacity statement identifies the constrained object, the family of allowed inputs, the desired output and the approximation tolerance. It then explains which components can bypass the constraint and which cannot. For attention, that means separating low-dimensional scores from normalized probability patterns, values, masks and residual composition. It also means distinguishing exact algebra from finite-precision usefulness.

The central result of our construction is deliberately modest: rank-one scores can yield full-rank attention, while full-rank attention can yield indistinguishable outputs. Those two facts are compatible with meaningful architectural bottlenecks. They simply show that rank alone is not a certificate of the property a reader may care about. Future work can improve diagnostics by aligning them with discriminating tasks and by linking approximation errors through the complete block. A promising report would explain not only that a spectrum changed, but which recoverable associations were gained or lost, at what compute cost, and under which precision assumptions. That would turn a convenient matrix statistic into a scientifically interpretable account of model behavior.

A practical acceptance criterion should therefore state a target contrast and an allowable output error before inspecting spectra. This prevents a threshold from being adjusted after observing the result. It also makes negative findings useful: a method may retain exact rank while exceeding the task's error tolerance, or reduce rank while remaining adequate for a low-dimensional summary. Those outcomes should be reported as different behaviors, not forced into a common success label.

Core equations and definitions

Constructed analytical figures

Comparison and reporting contract

Evaluation protocol

• Measure scores, normalized attention, values and outputs separately.
• Specify masks, precision, numerical-rank threshold and value scale.
• Use exact negative controls before neural runs.
• Match budgets and retain training failures.

Limitations

• No neural models are trained.
• The two-token symmetric family is not universal attention.
• Source proofs are not independently audited.

Open research questions

• Which task-aligned rank diagnostics predict failed association recall?
• How do approximation errors accumulate through residual blocks?
• When is numerical rank more informative than exact rank?

Evidence and its boundaries

Introduces attention-based Transformer sequence architecture.

[1] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Identifies a head-dimension bottleneck under its stated model.

[2] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Studies rank degeneration in pure attention and distinctions introduced by residual and MLP components.

[3] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Establishes approximation results under explicit function-class assumptions.

[4] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Uses random features for efficient attention-kernel approximation.

[5] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Develops linear attention with recurrent evaluation.

[6] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.