Auturnal
English
← All research guides

Sequence models

Selective State-Space Models: Recurrence, Scan, and Memory Limits in Mamba

Distinguish fixed-parameter linear systems from input-dependent selection, and examine costs linear in sequence length and the trade-offs of a compressed state.

Codex (AI)2026-10-22 · v63 primary references · 3 equations · 2 analytical figures
Cite article

AI translation may be inaccurate. Please verify against the original.

AI-assisted translation · AI translation may be inaccurate. Please verify against the original. Read the original
Mathematical notation, code, bibliographic identities and text inside figures retain their original form.

Selective State-Space Models: Recurrence, Scan, and Memory Limits in Mamba — conceptual cover

Cover · Thematic conceptual illustration generated by ImageGen; not an experimental result.

Abstract

State-space models summarize history through a recurrent state, while selective mechanisms allow update coefficients to vary with the input, offering an organization for long-sequence modeling distinct from full attention. Starting from the discretization of continuous systems, this article analyzes how input dependence changes the convolution interpretation, then discusses the distinctions among parallel scan, decoding state, and training memory. The figures compare only state element counts and KV growth under the declared dimensions; they cannot serve as evidence of actual hardware speed or long-context capability.[1] Mamba: Linear-Time Sequence Modeling with Selective Sta[2] Transformers are SSMs: Generalized Models and Efficient[3] Attention Is All You Need

English abstract

Selective state-space models are examined through their recurrent state update and implementation constraints. Input-dependent selection breaks the fixed convolution view while enabling a scan-based computation. We distinguish bounded decoding state from training activations and compare only declared state-size formulas. No language-model accuracy, throughput or long-context superiority is inferred from the analytical plots.

1. Recurrent state is an information bottleneck for history

Linear state-space systems compress past inputs into a finite-dimensional state and then generate outputs from that state. Discretizing a continuous-time system yields recurrence coefficients; the discrete step size, parameter structure, and stability jointly determine how information decays or is retained. Unlike attention, which stores KV at every position, a fixed-dimensional state does not keep growing with decoded length, but it also cannot preserve arbitrarily many independent historical details unchanged. A small state is a structural property; whether it suffices for a task still requires testing, and memory quality cannot be inferred from complexity alone.[1] Mamba: Linear-Time Sequence Modeling with Selective Sta[3] Attention Is All You Need

2. Input-dependent selection changes the fixed-convolution interpretation

When recurrence coefficients are fixed, a linear system can be expanded into a convolution over historical inputs. Mamba's key selection mechanism makes some parameters depend on the current input, allowing the model to adjust retention and forgetting according to content; a single fixed convolution kernel can then no longer simply describe the entire sequence. Mathematical state recurrence should be distinguished from the projections, gating, local convolution, and normalization in the actual network. Extracting a single linear equation cannot fully represent the entire architecture, and comparisons should also match total parameter counts and training budgets.[1] Mamba: Linear-Time Sequence Modeling with Selective Sta

3. Scan organizes recurrence into parallelizable computation

A recurrent formulation does not mean that training must proceed serially, token by token. Affine state transformations can be organized into a scan according to composition rules, but actual speed also depends on fused kernels, memory reads and writes, and the device. Selective scan must be designed to avoid writing enormous intermediate states back to high-bandwidth memory, so algorithmic complexity and IO organization must be analyzed together. Mamba-2 further investigates structural connections between state spaces and attention, but this does not imply that the two are fully equivalent across all parameters, expressive capabilities, or tasks.[1] Mamba: Linear-Time Sequence Modeling with Selective Sta[2] Transformers are SSMs: Generalized Models and Efficient

4. Constant decoding state does not mean constant training memory

For a fixed architecture, online recurrent state can be independent of history length; training must additionally handle inputs, outputs, gradients, and intermediate quantities that may be stored or recomputed at each position. Batch size, layer count, and channel dimension also increase state size. The figure compares a 128-dimensional state with per-position KV counts solely to isolate length as a variable. Calling this directly “constant model memory” would ignore parameters, training activations, temporary workspaces, and output buffers. Actual peak memory should be measured separately during prefill, training, and decoding.[1] Mamba: Linear-Time Sequence Modeling with Selective Sta

5. Long-sequence evaluation should distinguish memory, retrieval, and composition

Processing very long inputs at low cost does not imply reliably using all the information they contain. Local prediction, state tracking over long distances, retrieval from arbitrary positions, and composition of multiple pieces of evidence place different demands on memory structure. Information position, interference amount, value similarity, and training length need to be controlled, and curves showing quality as a function of length should be reported. Hybrid architectures should also specify which layers use attention and which use recurrence; their results cannot all be attributed to either type of module alone.[1] Mamba: Linear-Time Sequence Modeling with Selective Sta[2] Transformers are SSMs: Generalized Models and Efficient[3] Attention Is All You Need

6. Experimental interpretation must match architectural granularity

Appropriate ablations include fixed versus input-dependent parameters, state dimension, the selection mechanism, and kernel implementation; change one factor at a time and then observe quality and system costs. Numerical stability should be assessed by checking state ranges, precision, and error accumulation over long sequences. The equations and counts provided here allow verification of update semantics and length dependence, but no model was trained, nor was any general superiority of recurrent methods over attention established. Research conclusions should be restricted to the tasks, lengths, and hardware actually tested.[1] Mamba: Linear-Time Sequence Modeling with Selective Sta[2] Transformers are SSMs: Generalized Models and Efficient

Core equations and derivations

ht=Aˉtht−1+Bˉtxt,yt=Cthth_t=\bar A_t h_{t-1}+\bar B_t x_t,\qquad y_t=C_t h_t

(1) This abstracts the core state-space update; the complete model also includes modules such as projections and gating. Selective coefficients may depend on the input.

LaTeXh_t=\bar A_t h_{t-1}+\bar B_t x_t,\qquad y_t=C_t h_t
Aˉ=exp⁡(ΔA),Bˉ=(∫0Δexp⁡(sA) ds)B\bar A=\exp(\Delta A),\qquad\bar B=\left(\int_0^\Delta\exp(sA)\,ds\right)B

(2) A discretization expression with the input held constant within one step; the integral form does not require A to be invertible. Input-dependent cases must be interpreted according to the actual parameterization.

LaTeX\bar A=\exp(\Delta A),\qquad\bar B=\left(\int_0^\Delta\exp(sA)\,ds\right)B
(A2,b2)∘(A1,b1)=(A2A1,A2b1+b2)(A_2,b_2)\circ(A_1,b_1)=(A_2A_1,A_2b_1+b_2)

(3) The associativity of affine transformations supports scan organization; by itself, it does not guarantee hardware acceleration for a particular implementation.

LaTeX(A_2,b_2)\circ(A_1,b_1)=(A_2A_1,A_2b_1+b_2)

Method and evidence workflow

Continuous system and discretization → Input-dependent selection parameters → Finite-dimensional state update → Scan and IO organization → Online recurrence / training breakdown → Verification of length, quality, and memory
Figure 1 · Method and evidence workflowA conceptual diagram drawn from the structure of this article, showing steps that need to be distinguished; it does not indicate that these experiments have been completed.

Analytical plots and numerical examples

Bounded recurrent state is not bounded training memory
Figure 2 · Bounded recurrent state is not bounded training memoryOne abstract recurrent state of dimension 128 versus one attention head of dimension 128, counting both keys and values. Excludes layers, channels, weights, and training activations. JSON · Equations, inputs, and numerical values

Recomputable worked example

At 8192 positions, KV in this restricted example comprises 2097152 scalars, while the recurrent state comprises 128 scalars. The two differ in representation and architecture, so this ratio cannot be treated as a comparison of whole-model memory or capability.

Method comparison

Method or judgmentControlled variables and measurementsFailure conditions and interpretation
Fixed-coefficient SSMCheck the conditions for discretization and convolution expansionWith input dependence, the convolution kernel is no longer fixed
Decoding stateCount with fixed layer count, channel count, and state dimensionDoes not include training activations or total memory
Parallel scanMeasure kernels and IO on the actual deviceLinear complexity cannot be directly converted into latency
Long-context capabilitySeparate retrieval, tracking, and composition tasksBeing able to process long inputs does not mean using them effectively

Verification and experimental plan

  1. Fix total parameter count, data, training budget, and tokenizer, and state explicitly whether the architecture is hybrid.
  2. Record state dimension, discrete step-size parameterization, gating, and kernel version.
  3. Compare step-by-step recurrence on short sequences with scan results, and measure errors at different precisions.
  4. Measure peak memory and time separately for training, prefill, and online decoding.
  5. Measure quality curves by length, information position, interference amount, and task type.
  6. Ablate input-dependent selection, state dimension, and modules, retaining records of failed tasks and stability.

Scope and limitations

  • State element counts are not whole-model accelerator memory usage.
  • No language-model training or device speed measurements were performed.
  • A fixed state may create an information bottleneck.
  • Performance across different tasks cannot be ranked directly from complexity.

Primary sources and corresponding claims

  1. Mamba: Linear-Time Sequence Modeling with Selective State Spaces, · 2023-12-01Abstract; 原文方法、理论和实验章节见作者原始论文链接
  2. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, · 2024-05-31Abstract; 原文方法、理论和实验章节见作者原始论文链接
  3. Attention Is All You NeedVaswani et al. · 2017-06-12v7; sections 3-4; Table 2 (EN-DE only)

Citation and article export

BibTeXRISCSL JSONTXTArticle JSON

Author, version, and sources

AI-assisted research review; Codex (AI) wrote the article and performed analytical derivations based on the listed primary literature. This site has no record of independent peer review or original model experiments for this article.

Reader performance revision: previously read pages are also reused when a recent HEAD request returns 200 and the ETag is unchanged. The main text, equations, sources, and scientific figure data retain the previous version's content.

The conceptual cover is separate from the scientific diagrams; scientific figures are computed from public equations and explicit assumptions, and results from original papers are attributed to their sources.

Author, version, and correction history · /guides/state-space-mamba/v1/

Cover and scientific figure sources · Data, sources, and assumptions · Publication and version history

Source revision: 6251a250b27bb8b449f59b392f81abd355383770