What a Recurrent State Can Remember: Decay, Selectivity, and Sequence Memory

1. Memory is a contract with a future query

A sequence model's memory cannot be evaluated without specifying what a future query may ask. Remembering a running average, the most recent event, every key-value association or an exact historical string are different tasks. A compact state can be sufficient for one and insufficient for another. Describing a recurrent architecture as having long memory without naming the query distribution therefore leaves the central scientific question unresolved.

This review studies retention and access through two small constructions. A scalar leaky accumulator shows the tradeoff between retaining an old signal and accumulating later interference. A selective-writing control shows how a gate can improve retention when it knows which inputs matter, while making the quality of that selection explicit. These examples do not represent the full nonlinear dynamics of a learned state-space model. They isolate mechanisms that otherwise become entangled in benchmark comparisons. The broader argument is that sequence length, state dimension and runtime complexity cannot substitute for a measurement of which information remains recoverable, under what noise and precision assumptions, and by which readout.

2. What the primary records establish

S4 develops efficient structured state-space sequence modeling. HiPPO formulates recurrent compression through polynomial projections under a specified importance measure. Mamba introduces input-dependent selective state spaces, while the structured state-space duality work connects families of state-space and attention computations. Their mathematical objects and computational procedures differ, so no single scalar recurrence should be presented as their exact implementation.

Hungry Hungry Hippos investigates sequence-modeling capabilities including recall and token comparison. Zoology studies associative recall in efficient language models and reports a diagnostic relationship to its measured language-modeling gaps. These papers motivate asking what information an architecture retains and how it uses that information. Our inspection covers the original abstracts and bibliographic records; the source implementations, proofs and training runs are not independently reproduced. Historical speed or quality comparisons are not used as current universal rankings. The constructed analysis below is original and deliberately narrower. It uses elementary recurrence and variance calculations to distinguish memory mechanisms, then identifies empirical controls that would be needed to evaluate learned architectures fairly.

Sources: [1], [2], [3], [4], [5], [6]

3. A state is not a transcript

For a recurrence h_t=F(h_{t-1},x_t), the current state is a function of the entire observed prefix, but that does not make it a recoverable transcript. Several prefixes may map to the same state or to states indistinguishable under numerical precision and the available readout. The relevant property is sufficiency for the intended future query, not merely dependence on the past.

A running sum is sufficient for queries about the sum and generally insufficient for queries about the order of addends. A state that stores the most recent matching key may answer recency queries but lose earlier versions. These are not necessarily defects; compression is useful precisely when irrelevant distinctions can be discarded. The difficulty arises when a benchmark or deployment changes which distinctions are relevant. A model trained on mostly local predictions may compress away rare distant associations. A good memory evaluation therefore specifies both the sequence-generating process and the query family. It also includes a control task where compression should help, so that maintaining every detail is not mistakenly treated as the only valid objective.

4. The scalar leaky accumulator

Consider h_t=rho h_{t-1}+x_t with initial state zero and zero less than rho less than one. A unit signal written at time zero contributes rho raised to k after k further steps. The impulse response decays geometrically, and the half-life is log(1/2) divided by log(rho). Larger rho extends retention of that signal.

However, subsequent inputs also enter the state. If they are independent, mean-zero noise with variance sigma squared, their accumulated variance after k steps is sigma squared times the sum of rho raised to twice each lag from zero to k-1. This is a finite geometric sum. The model therefore contains two effects: the signal decays, while interference accumulates. Figure one plots their combined signal-to-noise ratio for several rho values at a fixed noise variance. It is not a measured accuracy curve. The construction makes assumptions visible and permits exact numerical checks, including the limiting case rho equal to one, where the signal remains but noise variance grows linearly with the number of later writes.

5. Retention and interference cannot be merged

A large retention coefficient can preserve an old impulse, yet also preserve every subsequent disturbance. In the scalar model, signal-to-noise ratio after k steps equals rho to the power 2k divided by the accumulated noise variance. For rho less than one, old-signal energy eventually decays exponentially. For rho equal to one, it remains constant but is diluted by linearly growing noise variance.

This does not establish a universally optimal rho for neural sequence models. A nonlinear network can distribute information across dimensions, gate writes, rescale states or use query-dependent decoding. The example instead shows why an impulse-response plot alone is insufficient evidence of usable memory. A long tail can coexist with poor recovery in a noisy stream. An experiment should vary the number and amplitude of distractors independently from elapsed sequence length. It should also distinguish noise that is statistically independent from structured distractors sharing keys or syntax with the target. A model may handle one and fail on the other. The memory claim must identify the interference process under which retention is useful.

6. State dimension changes the space of summaries

A vector state can store several timescales or features rather than a single accumulator. In a diagonal linear recurrence, each coordinate has its own decay factor and input projection. The resulting memory kernel is a combination of modes. This can approximate a richer family of summaries, but the number of dimensions alone does not specify which summaries are learned or whether the readout can use them.

The comparison should distinguish dimensions allocated to stable storage from dimensions carrying transient computation. It should also state the numerical precision and dynamic range of the state. An exact real number can encode unbounded information in a mathematical construction, but such encodings are not robust under finite precision. Conversely, counting raw bits gives only a storage ceiling, not an achievable task capacity. A practical study can probe recoverable associations as a function of state size while keeping training and readout budgets explicit. A sudden plateau may indicate interference, optimization, precision or decoding limits. Without additional controls, it should not be attributed automatically to the architecture's nominal number of state coordinates.

7. Selective writing introduces an information assumption

Modify the scalar recurrence so that only inputs selected by a binary gate are written. If the gate accepts the initial signal and rejects every later distractor, the interference term vanishes. This ideal control demonstrates the value of selection, but it also assumes that the gate knows what the future query will require or can infer relevance from the current input.

That assumption is often the difficult part of a sequence task. A token that looks irrelevant when encountered may become important after a later query. A gate optimized for frequent queries can discard information required by rare ones. Selectivity therefore moves part of the memory problem into a relevance-prediction problem. Figure two varies the probability of admitting distractors while keeping retention fixed. Under independent Bernoulli gating of mean-zero noise, the interference variance is multiplied by the admission probability. The curve is an expectation under the declared model, not a claim about learned gate accuracy. It shows precisely which assumption makes the benefit appear and what measurement would be needed to justify transferring it to a trained system.

8. Forgetting can be the correct behavior

A memory system should not preserve obsolete or contradictory information indefinitely when the task asks for the current state of the world. If a key is updated, retaining all old values without a temporal policy can make retrieval ambiguous. A controlled benchmark should therefore distinguish immutable associations, last-write-wins updates and queries about historical versions.

These variants require different sufficient statistics. A fixed-size state may solve one well while failing another for principled reasons. Reporting them together as one recall score can hide the distinction. The same issue arises in continual dialogue: a later correction may supersede an earlier statement, while an explicit historical question may require both. Evaluation should specify the semantics of updates and include tests that distinguish forgetting from failed retention. An architecture that discards stale information may be better aligned with the intended task than one that returns an old association faithfully. Our scalar construction has no semantic update rule. It is useful only as a retention and interference control, and should not be used to rank models on temporal reasoning.

9. Access is separate from storage

Information can remain encoded in a state without being accessible to the deployed readout. A powerful probe trained after the fact may recover a historical label, while the model's actual next-token head fails to use it. That result demonstrates recoverability under the probe's supervision and capacity, not successful use by the original model.

Conversely, a simple probe's failure does not prove that information is absent. It may be stored nonlinearly or in a distributed form. A rigorous report distinguishes at least three measurements: task performance with the original readout, recovery with a specified diagnostic probe, and sensitivity of the output to controlled changes in the historical input. Each answers a different question. Probe training data must be separated from evaluation data, and probe capacity should be reported. If a probe effectively learns the entire task from side information, its success can be misinterpreted as memory. Counterfactual prefix pairs help: change only the historical fact, preserve later distractors, and test whether the model or probe responds appropriately to that change.

10. Parallel training does not imply random access

A recurrence may admit an efficient parallel scan during training while still using a compact state during autoregressive inference. Conversely, an attention model may store a growing cache and access earlier representations explicitly. These computational organizations have different memory and latency costs, but neither cost pattern alone determines task quality.

The comparison must state whether it concerns full-sequence training, prompt processing or incremental generation. A kernel optimized for one phase may not dominate another. Batch size, sequence length and state precision can change hardware utilization substantially. Reporting asymptotic complexity without the served workload can therefore obscure the practical tradeoff. It is also misleading to infer from a parallel algorithm that the inference state retains every token independently. Parallel evaluation is a way to compute a function, not a proof of the function's information capacity. A fair memory study combines behavioral tasks with measured resource use in each phase, and avoids interpreting throughput as evidence that historical information is either preserved or lost.

11. A finite-precision perspective

Suppose a state uses B bits in total and a query can request one of many independently chosen historical values. A counting argument can show that exact recovery of every possible transcript requires enough distinct states to distinguish those transcripts. This is an information-theoretic ceiling under the exact-recovery requirement, not a lower bound on ordinary language modeling.

Real tasks often contain redundancy, restricted queries and tolerance for error, so the required state can be much smaller. Learned models may exploit those regularities effectively. The important point is to state which simplification makes compression possible. A benchmark that samples many independent random associations stresses a different regime from natural text with repeated semantic structure. Both are valuable, but neither should stand in for the other. Precision experiments can further reveal whether apparent capacity depends on fragile amplitudes or robust separations. One can quantize the stored state at inference while keeping the readout fixed, then measure which tasks degrade. Such an intervention must be labeled as a changed numerical implementation rather than a pure architectural comparison.

12. Designing a memory stress test

Construct sequences with independently controlled target age, distractor count, distractor variance and update semantics. Include both random associations and structured text-like patterns. Ask queries whose answers depend on the target, and provide a no-target control to detect guessing or leakage from the query format. Vary state dimension and precision while matching the rest of the training budget.

Report accuracy by target age rather than only an aggregate average. Include confidence intervals over independently sampled sequences and retain failed training seeds. If the model uses an external cache or retrieval component, account for that storage explicitly. To separate storage from access, compare the original readout with a carefully specified probe, but do not replace the primary task result with the best probe score. A useful test suite also includes tasks where forgetting is desirable. The resulting profile is more informative than one headline long-context number. It can show whether a model is strong at summaries, recent updates, distant exact associations or query-dependent composition, and where its efficient state representation imposes a practical tradeoff.

13. The figures and their assumptions

The first figure uses a unit initial signal, independent zero-mean distractors of unit variance and selected retention coefficients. Its vertical axis is the analytical signal-energy-to-noise-variance ratio. At zero elapsed steps the interference variance is zero, so the ratio is undefined; that point is intentionally excluded rather than plotted as an arbitrary large number. The program checks the finite geometric sum directly.

The second figure fixes a retention coefficient and varies distractor admission probability under independent Bernoulli gates. It reports expected interference variance at a fixed age. The gate is assumed always to admit the original signal, and its decisions are independent of the noise values. If gating depends on noise amplitude or sign, the mean and variance calculation changes. That limitation is part of the caption and the executable model. These explicit assumptions are the scientific content of the illustrations. The curves cannot be read as measurements of a particular selective state-space architecture. Their value is that a reader can identify exactly which mechanism was changed and reproduce every numerical point.

14. What the checks prove and leave open

The companion code compares recurrence simulation with closed-form impulse decay, verifies the noise-variance sum for several lengths, and checks the rho-equals-one limit separately. It verifies that independent gate admission scales the variance by its probability, and tests that perfect rejection removes interference in the declared model. The checks use deterministic algebra and finite enumerations rather than random training.

Passing them confirms consistency among the text, equations and plotted data. It does not establish that a neural model learns the ideal gate, that its state behaves linearly, or that its readout achieves the corresponding signal recovery. It also does not verify the source papers' empirical comparisons. These are distinct evidence levels. A deployment decision would require task-specific learned-model experiments and resource measurements, including behavior under unexpected queries and updates. By keeping the small-model verification separate, the article offers a reusable diagnostic without exaggerating its significance. The unanswered question is not whether the arithmetic works, but how much of this simplified mechanism survives in a trained architecture under a realistic workload.

15. A useful comparison is a profile

Sequence architectures should be compared through a profile of tasks and budgets rather than a single memory label. The profile can distinguish exact association recall, compressed summary, temporal update, compositional query and streaming prediction. It can report how performance changes with state size, precision, target age and distractor structure.

Such a profile permits meaningful tradeoffs. A model may sacrifice arbitrary exact recall to gain efficient streaming summaries, which can be appropriate for its application. Another may justify a larger cache because queries require many independent historical details. The choice becomes clearer when the evaluation exposes those requirements explicitly. Our constructions explain why a decay constant is not a complete memory measure and why selectivity is only beneficial under a relevance assumption. They also show how apparently favorable retention can coexist with interference. Future work should connect these simple mechanisms to learned internal dynamics while resisting the temptation to equate a suggestive probe with a complete causal explanation. The goal is an auditable account of what information survives and what the model can actually do with it.

16. Limits of the present review

This article does not introduce a new sequence architecture or train a model. Its source synthesis is selective, and its original calculations concern linear scalar memory with elementary noise assumptions. Nonlinear interactions, multidimensional coding, normalization, adaptive discretization and learned query dependence can alter the behavior substantially. Those omissions limit the reach of the conclusions.

They do not make the controls irrelevant. A proposed explanation of long memory should at least distinguish retention from interference, storage from access, and state size from task sufficiency. The finite models make those distinctions concrete and reproducible. A stronger future study would instrument trained models, formulate falsifiable predictions from their measured state dynamics, and test those predictions on held-out query families. It would publish cases where the explanation fails as well as cases where it succeeds. That approach can move beyond architecture slogans toward a quantitative science of sequence memory, while preserving the practical advantages that make efficient recurrent models worth studying.

Core equations and definitions

Constructed analytical figures

Comparison and reporting contract

Evaluation protocol

• Vary target age and distractor count separately.
• Test immutable, current-value and historical-version queries.
• Report readout and probe performance separately.
• Account for state precision and any external cache.

Limitations

• No source architecture is implemented or benchmarked.
• The recurrence is scalar and linear.
• Gating is independent and relevance is assumed, not learned.

Open research questions

• Which learned states preserve rare query-relevant information?
• How does finite precision affect recoverable association capacity?
• When does selective forgetting improve temporal reasoning?

Evidence and its boundaries

Develops efficient structured state-space sequence models.

[1] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Formulates recurrent compression using polynomial projections.

[2] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Introduces input-dependent selective state spaces.

[3] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Connects state-space and attention computations through structured matrices.

[4] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Investigates recall and comparison capabilities in state-space language modeling.

[5] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.

Studies associative recall as a diagnostic in efficient language models.

[6] — Original arXiv bibliographic record and abstract inspected on 2026-10-28; full proof and source experiments not independently audited. Inspected: Primary arXiv abstract and bibliographic record.