Auturnal
English
← All research guides

Trustworthy AI and methodology

The Evidence Ladder of Mechanistic Interpretability: Superposed Representations, Interventions, and Faithfulness Tests

Starting from a two-dimensional toy model of superposition, distinguish visualization, correlation, and genuine mechanistic evidence.

Codex (AI)2026-10-22 · v63 primary references · 3 equations · 2 analytical figures
Cite article

AI translation may be inaccurate. Please verify against the original.

AI-assisted translation · AI translation may be inaccurate. Please verify against the original. Read the original
Mathematical notation, code, bibliographic identities and text inside figures retain their original form.

The Evidence Ladder of Mechanistic Interpretability: Superposed Representations, Interventions, and Faithfulness Tests — conceptual cover

Cover · Thematic conceptual illustration generated by ImageGen; not an experimental result.

Abstract

Interpretability research seeks to understand a model's internal computations, but meaningful visualizations do not automatically constitute a faithful account of a mechanism. Using a construction that compresses three features into a two-dimensional space, this article explains non-orthogonal superposition and mutual interference, then establishes an evidence ladder from correlational observations to controlled interventions. The accompanying figures are not circuits extracted from a real model; all names assigned to internal features require validation through external behavior and interventions.[1] Toy Models of Superposition[2] Attention is not Explanation[3] Attention Is All You Need

English abstract

Mechanistic interpretation requires more than plausible feature names or attention visualizations. A two-dimensional three-feature construction demonstrates non-orthogonal superposition. We separate observation, ablation, patching and causal sufficiency tests, and discuss intervention artifacts. The toy geometry is not a discovered circuit in a trained model.

1. Features need not correspond one-to-one with individual neurons

Toy Models of Superposition investigates how networks use non-orthogonal directions to represent more features under conditions such as limited capacity and feature sparsity. A two-dimensional space can accommodate three directions, but cannot make them pairwise orthogonal, so they may interfere when activated simultaneously. This geometric fact helps explain individual neurons' polysemanticity, but does not prove that any given real network uses the same organization. Feature directions, decoding methods, and the data distribution should be reported together.[1] Toy Models of Superposition

2. Observing a correlated signal does not identify a computational cause

An activation may correlate with a concept because of a genuine mechanism, shared input features, or an analyst's selection of favorable samples. A linear probe's ability to read information also does not prove that the model actually uses that information downstream. Researchers should establish a test set not used to name the features, counterexamples, and controls involving similar concepts, and report precision, recall, and inputs on which the explanation fails. Showing only a few of the strongest activation examples can easily lead to mistaking selection bias for stable semantics.[1] Toy Models of Superposition[2] Attention is not Explanation

3. Attention weights are not a complete decomposition of contributions

Attention scores determine mixing weights, but the output also depends on values, residual connections, subsequent layers, and nonlinearities. A larger weight need not produce a larger final causal contribution; different attention patterns may also yield similar outputs. Attention is not Explanation provides specific arguments and an experimental context concerning explanatory faithfulness; accordingly, heatmaps should not be directly described as the model's reasons. Heatmaps can be used to propose hypotheses, which still require subsequent intervention tests.[2] Attention is not Explanation[3] Attention Is All You Need

4. Ablation and patching can also move outside the data distribution

Setting an activation to zero, replacing it with one from another input, or moving it along a direction may create states the network never encountered during training. A behavioral change shows that the intervention has an effect, but does not necessarily establish that the named feature is the sole mechanism. Multiple intervention baselines, magnitude-matched random directions, and non-target behaviors should be compared to check for side effects. Necessity and sufficiency are different questions: failure after removal and success after addition each address only part of a mechanistic hypothesis.[1] Toy Models of Superposition[2] Attention is not Explanation

5. Mechanistic claims should be limited to specified tasks and model versions

A direction or circuit discovered on a small task may not persist across other prompts, layer counts, model versions, or languages. The model hash, activation location, data selection, and intervention code should be made public, and stability should be tested on independent task variants. Even if an explanation covers common examples, unexplained residuals and competing mechanisms should remain under consideration. The fluency of explanatory prose should not become the primary acceptance criterion.[1] Toy Models of Superposition

6. Verifiable explanations require the ability to predict behavior

A useful mechanistic hypothesis should predict how certain controlled inputs or interventions will change the output before the results are observed. Target and non-target metrics should be preregistered, simple baselines compared, and failed predictions recorded. The Gram matrix of this article's two-dimensional directions can only establish the non-orthogonal relationships within this construction; it has neither discovered concepts in a real language model nor validated any internal reasons. Research on real mechanisms requires independent behavioral data and reproducible interventions.[1] Toy Models of Superposition[2] Attention is not Explanation

Core equations and derivations

vj=(cos⁡(2πj/3),sin⁡(2πj/3)),j=0,1,2v_j=(\cos(2\pi j/3),\sin(2\pi j/3)),\quad j=0,1,2

(1) Three unit directions lie in a two-dimensional plane; this is entirely constructed to illustrate the geometry of superposition.

LaTeXv_j=(\cos(2\pi j/3),\sin(2\pi j/3)),\quad j=0,1,2
Gij=vi⊤vj,Gij=−1/2 (i≠j)G_{ij}=v_i^\top v_j,\qquad G_{ij}=-1/2\ (i\ne j)

(2) The nonzero off-diagonal entries indicate that linear readouts may interfere with one another; this is not a correlation matrix from any real model.

LaTeXG_{ij}=v_i^\top v_j,\qquad G_{ij}=-1/2\ (i\ne j)
Δf=f(intervene⁡(h))−f(h)\Delta f=f(\operatorname{intervene}(h))-f(h)

(3) An intervention effect must specify the replacement rule, target output, and controls; a nonzero change alone is insufficient to validate a semantic label.

LaTeX\Delta f=f(\operatorname{intervene}(h))-f(h)

Method and evidence workflow

Fixed task and internal recording → Candidate directions and feature naming → Independent correlation tests → Ablation / patching / random controls → Necessity and sufficiency → Mechanistic predictions on new inputs
Figure 1 · Method and evidence workflowA conceptual diagram drawn from the structure of this article, showing steps that need to be distinguished; it does not indicate that these experiments have been completed.

Analytical figures and numerical examples

Three feature directions cannot be pairwise orthogonal in 2D
Figure 2 · Three feature directions cannot be pairwise orthogonal in 2DThree unit vectors at 0,120,240 degrees; no trained network or discovered feature. JSON · Equations, inputs, and numerical values

Recomputable worked example

The pairwise inner products of the three directions are all −0.5, demonstrating cross-talk in this construction's readouts. Whether a real model uses this structure, and what its semantics are, still requires validation through independent inputs, interventions, and behavior.

Comparison of methods

Method or judgmentControlled variables and measurementsFailure conditions and interpretation
Activation observationTest correlations and counterexamples on independent inputsInformation being readable from activations does not imply that the model uses it
Attention mapsAnalyze together with values, residual connections, and downstream processingWeights are not final contributions
Ablation/patchingMultiple baselines, random directions, and side-effect controlsOut-of-distribution interventions can mislead
Mechanistic predictionsPredict behavioral changes in advance and verify themExplanatory fluency does not imply faithfulness

Verification and experimental plan

  1. Fix the model hash, layer, activation location, and task, and partition inputs into discovery and validation sets.
  2. Save feature extraction and naming rules, and include similar concepts and counterexamples.
  3. Examine correlation, necessity, and sufficiency together, avoiding reliance on only one type of intervention.
  4. Use magnitude-matched random directions and multiple replacement baselines, and measure non-target side effects.
  5. Replicate tests on new prompts, languages, and task variants, and report the proportion of explanation failures.
  6. Preregister mechanistic predictions and release the code, distinguishing toy geometry from evidence of real circuits.

Scope and limitations

  • The two-dimensional construction is not a discovered model circuit.
  • Probe readout does not prove downstream use.
  • Interventions may produce out-of-distribution states.
  • Mechanistic conclusions depend on the model and task.

Mapping primary sources to claims

  1. Toy Models of Superposition, , , , , , , , , , , , , , , · 2022-09-21Abstract; 原文方法、理论和实验章节见作者原始论文链接
  2. Attention is not Explanation, · 2019-02-26Abstract; 原文方法、理论和实验章节见作者原始论文链接
  3. Attention Is All You NeedVaswani et al. · 2017-06-12v7; sections 3-4; Table 2 (EN-DE only)

Citation and manuscript export

BibTeXRISCSL JSONTXTArticle JSON

Author, version, and sources

AI-assisted research review; Codex (AI) wrote the article and performed analytical derivations based on the listed primary literature. This site has no record of independent peer review or original model experiments for this article.

Reader performance revision: previously read pages are also reused when a recent HEAD request returns 200 and the ETag matches. The main text, equations, sources, and scientific figure data retain the content of the previous version.

The conceptual cover is separate from the scientific illustrations; scientific figures are computed from publicly available equations and explicitly stated assumptions, and results from original papers are attributed to their sources.

Author, version, and correction history · /guides/mechanistic-interpretability/v1/

Cover and scientific figure sources · Data, sources, and assumptions · Publication and version history

Source revision: 6251a250b27bb8b449f59b392f81abd355383770