Auturnal
English
← All research guides

Sequence models

Positional Encoding and Long Context: Rotary Phase, Distance Bias, and Effective Information Use

Prove the relative-position inner-product relation, then distinguish length extrapolation from genuine retrieval and composition capabilities.

Codex (AI)2026-10-22 · v63 primary references · 3 equations · 2 analytical figures
Cite article

AI translation may be inaccurate. Please verify against the original.

AI-assisted translation · AI translation may be inaccurate. Please verify against the original. Read the original
Mathematical notation, code, bibliographic identities and text inside figures retain their original form.

Positional Encoding and Long Context: Rotary Phase, Distance Bias, and Effective Information Use — conceptual cover

Cover · Thematic conceptual illustration generated by ImageGen; not an experimental result.

Abstract

Positional encoding provides sequence order information, but accepting longer inputs does not imply reliably using more distant information. This article derives the relative-position inner product of RoPE, explains phase changes at different frequencies, and compares them with ALiBi's distance bias and local sparse attention. The accompanying plots show cosine phases at explicitly specified frequencies, not model accuracy; the verification plan separately measures length support, positional retrieval, and the composition of multiple pieces of evidence.[1] RoFormer: Enhanced Transformer with Rotary Position Emb[2] Train Short, Test Long: Attention with Linear Biases En[3] Longformer: The Long-Document Transformer

English abstract

We derive the relative-position inner product of rotary embeddings and contrast it with linear attention biases. Phase curves illustrate frequency behavior only. Context acceptance, retrieval and evidence composition are evaluated separately, with explicit training-length and truncation controls. No long-context accuracy or universal extrapolation guarantee follows from positional algebra.

1. How relative position emerges from absolute rotation

When the two-dimensional subspaces of queries and keys are each rotated according to position, the two absolute positions in the inner product can be combined into a rotation corresponding to their positional difference. This provides an algebraic structure for relative-position relations, but attention scores still depend on input content, frequencies, and learned representations. This identity does not prove that a model generalizes to arbitrary lengths, nor does it guarantee that different positional differences can all be uniquely and stably identified. Implementations should verify dimension pairing, rotation direction, positional offsets, and cached continuation, paying particular attention to indices after chunked prefill.[1] RoFormer: Enhanced Transformer with Rotary Position Emb

2. Multifrequency phase and extrapolation are not the same thing

Higher frequencies rotate more rapidly with distance, while lower frequencies change more slowly; a single frequency exhibits periodic similarity. Combining multiple frequencies can provide richer representations, but training covers only a limited range of positions and data relationships. Position scaling or frequency adjustment changes the phase distribution presented to the model and requires renewed validation. The three angular frequencies 1, 0.1, and 0.01, expressed in radians, selected for the plot serve only to illustrate scales of variation. They cannot determine a model's maximum context length, much less justify interpreting similar curves as inevitable semantic confusion.[1] RoFormer: Enhanced Transformer with Rotary Position Emb

3. Linear distance bias introduces another inductive preference

ALiBi adds a distance-dependent linear term to attention scores and assigns different slopes to different heads. Its extrapolation behavior differs from that of rotary encoding, but both depend on training and the task. A stronger preference for nearby positions may suit some language prediction tasks, while also making distant evidence harder to use. Comparisons should hold model size, training tokens, and evaluation length constant, and disclose the bias implementation and mask; one model's test results cannot be generalized into a ranking of positional schemes across all tasks.[2] Train Short, Test Long: Attention with Linear Biases En

4. Sparse connections change reachable paths and computation

Local windows or global positions can reduce the cost of full attention while also changing the paths information must traverse to propagate from one position to another. Information outside the window may not be directly reachable within a single layer; increasing the number of layers, global tokens, and task structure affect the outcome. Positional encoding represents positions, whereas the sparsity pattern determines which interactions are allowed; the two should be ablated separately. A mask that theoretically supports long inputs does not automatically guarantee that critical information is effectively combined within the actual number of layers.[3] Longformer: The Long-Document Transformer

5. Long-context validation requires three levels

First, test whether the service accepts and correctly retains the specified length, ruling out silent truncation; next, test evidence retrieval at specified positions; finally, test reasoning or contradiction detection across multiple positions. Tests should sweep evidence positions, distractor counts, and distractor similarity, while controlling whether the answer can already be guessed from the model's general knowledge. A single successful needle retrieval supports only localization capability on that input distribution, not arbitrary long-document analysis. Reports should include the worst-performing positional interval and no-evidence controls.[1] RoFormer: Enhanced Transformer with Rotary Position Emb[3] Longformer: The Long-Document Transformer

6. Engineering problems can masquerade as model capability problems

Incorrect continuation positions in the KV cache, sliding-window truncation, tokenization differences, and batch padding can all change long-input results. First compare scores against a small reference implementation, then check consistency between continuous and chunked execution, and only afterward measure task quality. For any extrapolation modification, retain regression checks of capability at the original length and resource curves, rather than reporting only a few successful examples at the longest input length. This article provides positional algebra and analytical plots, not conclusions about length extension in actual models.[1] RoFormer: Enhanced Transformer with Rotary Position Emb[2] Train Short, Test Long: Attention with Linear Biases En[3] Longformer: The Long-Document Transformer

Core equations and derivations

(Rmq)⊤(Rnk)=q⊤Rn−mk(R_mq)^\top(R_nk)=q^\top R_{n-m}k

(1) Within the same family of two-dimensional rotations, R_m transposed times R_n equals R_(n−m); a complete implementation applies this to each subspace.

LaTeX(R_mq)^\top(R_nk)=q^\top R_{n-m}k
ϕj(Δ)=cos⁡(θjΔ)\phi_j(\Delta)=\cos(\theta_j\Delta)

(2) Shows the phase corresponding to the frequency, not attention weights, retrieval rate, or model quality.

LaTeX\phi_j(\Delta)=\cos(\theta_j\Delta)
sij=qi⊤kj−mh(i−j),j≤is_{ij}=q_i^\top k_j-m_h(i-j),\quad j\le i

(3) Illustration of linear distance bias in the causal setting; actual scaling and per-head slopes must be recorded according to the implementation.

LaTeXs_{ij}=q_i^\top k_j-m_h(i-j),\quad j\le i

Method and evidence workflow

Training length and position definitions → Rotation frequencies / distance bias → Connection mask and information paths → KV continuation and truncation checks → Position- and distractor-controlled tests → Separate retrieval, composition, and resource assessments
Figure 1 · Method and evidence workflowA conceptual diagram drawn from the structure of this article, showing steps that need to be distinguished; it does not indicate that these experiments have been completed.

Analytical plots and numerical examples

Multiple rotation frequencies evolve on different scales
Figure 2 · Multiple rotation frequencies evolve on different scalesThree declared scalar frequencies; one cosine component per frequency. Not a full RoPE representation or attention score. JSON · Equations, inputs, and numerical values

Recomputable worked example

At distance 32, the three cosine components are approximately 0.834, −0.998, and 0.949, respectively. They are only phases, neither model attention weights nor proof of retrieval errors or an upper bound on length.

Method comparison

Method or judgmentControlled variables and measurementsFailure conditions and interpretation
RoPEVerify rotation dimensions, frequencies, and continuation positionsThe algebraic relation does not guarantee generalization to unlimited lengths
ALiBiRecord per-head slopes and the causal maskPreference for nearby positions affects different tasks differently
Sparse attentionMeasure the window, global positions, and interlayer pathsPositional representation and connection structure must not be conflated
Long-document capabilityTest acceptance, retrieval, and composition separatelySingle-needle success cannot represent all long-document tasks

Verification and experimental plan

  1. Record training length, positional scheme, frequencies, tokenizer, and truncation strategy.
  2. Verify the rotary inner product and causal mask element by element on small tensors.
  3. Compare continuous execution, chunked prefill, and cached continuation to rule out indexing errors.
  4. Sweep length, position, distractors, and the number of evidence items to combine, and include no-evidence controls.
  5. Also check capability at the original length, GPU memory, latency, and the worst-performing positional interval.
  6. Ablate positional modifications separately from window size, layer count, and model size, and retain all failure cases.

Scope and limitations

  • A phase plot is not model accuracy.
  • A single-frequency example does not represent full RoPE.
  • No training experiments extending model context were conducted.
  • Retrieval capability cannot substitute for understanding multiple pieces of evidence.

Primary sources and corresponding claims

  1. RoFormer: Enhanced Transformer with Rotary Position Embedding, , , , , · 2021-04-20Abstract; 原文方法、理论和实验章节见作者原始论文链接
  2. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation, , · 2021-08-27Abstract; 原文方法、理论和实验章节见作者原始论文链接
  3. Longformer: The Long-Document TransformerBeltagy et al. · 2020-04-10v2 abstract and architecture description

Citation and article export

BibTeXRISCSL JSONTXTArticle JSON

Author, version, and sources

AI-assisted research review; Codex (AI) wrote the article and performed analytical derivations based on the listed primary literature. This site has no record of independent peer review or original model experiments for this article.

Reader performance revision: previously read pages are also reused when a recent HEAD response returns 200 and the ETag matches. The main text, equations, sources, and scientific figure data retain the previous version's content.

The conceptual cover is separate from the scientific illustrations; scientific figures are calculated from publicly available equations and explicitly stated assumptions, and all results from original papers are attributed.

Author, version, and correction history · /guides/positional-long-context/v1/

Sources for the cover and scientific figures · Data, sources, and assumptions · Publication and version history

Source revision: 6251a250b27bb8b449f59b392f81abd355383770