{
  "schema": "auturnal.deep-research-review.v1",
  "slug": "mechanistic-interpretability",
  "title": "机制可解释性的证据阶梯：叠加表示、干预与忠实性检验",
  "category": "可信AI与方法论",
  "dek": "从二维叠加玩具模型出发，区分可视化、相关性与真正的机制证据。",
  "english": "Mechanistic interpretation requires more than plausible feature names or attention visualizations. A two-dimensional three-feature construction demonstrates non-orthogonal superposition. We separate observation, ablation, patching and causal sufficiency tests, and discuss intervention artifacts. The toy geometry is not a discovered circuit in a trained model.",
  "kind": "ai_assisted_research_guide",
  "author": "Codex（AI）",
  "original_language": "zh-Hans",
  "published_date": "2026-10-22",
  "edition": 5,
  "version": 1,
  "previous_version": null,
  "refs": [
    "arxiv-2209.10652",
    "arxiv-1902.10186",
    "attention"
  ],
  "sections": [
    [
      "摘要",
      "可解释性研究试图理解模型内部计算，但有意义的可视化并不自动构成忠实机制。本文借助三特征压入二维空间的构造，解释非正交叠加及相互干扰，再建立从相关观察到受控干预的证据阶梯。配图不是从真实模型提取的回路；所有内部特征命名都需要外部行为和干预验证。",
      [
        "arxiv-2209.10652",
        "arxiv-1902.10186",
        "attention"
      ]
    ],
    [
      "1. 特征不必与单个神经元一一对应",
      "Toy Models of Superposition研究在容量有限与特征稀疏等条件下，网络如何用非正交方向表示更多特征。二维空间可容纳三个方向，但无法让它们两两正交，因此同时激活时可能干扰。这个几何事实有助于解释单神经元多义性，却不证明任意真实网络都采用同一种组织。特征方向、解码方法和数据分布需要一起报告。",
      [
        "arxiv-2209.10652"
      ]
    ],
    [
      "2. 观察到相关信号还没有找到计算原因",
      "激活与某个概念相关，可能因为真正机制、共同输入特征，或分析者选择了有利样本。线性探针能读取信息，也不证明模型下游确实使用该信息。应设置未用于命名特征的测试集、反例与相似概念对照，报告精确率、召回率和解释失败的输入。只展示几个最强激活样例，容易把选择偏差当作稳定语义。",
      [
        "arxiv-2209.10652",
        "arxiv-1902.10186"
      ]
    ],
    [
      "3. 注意力权重不是完整贡献分解",
      "注意力分数决定混合权重，但输出还依赖value、残差、后续层和非线性。较大的权重不必产生较大的最终因果贡献；不同注意力模式也可能得到相近输出。Attention is not Explanation提供了针对解释忠实性的具体论证与实验语境，应据此避免把热力图直接称作模型理由。热力图可以用于提出假设，随后仍需干预检验。",
      [
        "arxiv-1902.10186",
        "attention"
      ]
    ],
    [
      "4. 消融和补丁也可能离开数据分布",
      "将某个激活设零、替换为另一输入或沿方向移动，可能制造网络训练时从未见过的状态。行为变化说明干预有影响，但未必说明所命名特征是唯一机制。应比较多个干预基线、匹配幅度的随机方向和非目标行为，检查副作用。必要性与充分性是不同问题：删除后失败和加入后成功各自只回答部分机制假设。",
      [
        "arxiv-2209.10652",
        "arxiv-1902.10186"
      ]
    ],
    [
      "5. 机制主张应限定任务与模型版本",
      "一个小任务上发现的方向或回路，未必在其他提示、层数、模型版本或语种上保持。应公开模型哈希、激活位置、数据选择和干预代码，并在独立任务变体上测试稳定性。即便解释覆盖了常见样例，也应保留无法解释的残差和竞争机制。解释文本的流畅程度不应成为主要验收指标。",
      [
        "arxiv-2209.10652"
      ]
    ],
    [
      "6. 可复核解释需要行为预测能力",
      "一个有用机制假设应能在看见结果之前预测某些受控输入或干预会怎样改变输出。预先登记目标和非目标指标，比较简单基线，记录失败预测。本文二维方向的Gram矩阵只能证明该构造中的非正交关系，没有发现真实语言模型的概念，也没有验证任何内部理由。真实机制研究需要独立行为数据和可重复干预。",
      [
        "arxiv-2209.10652",
        "arxiv-1902.10186"
      ]
    ]
  ],
  "equations": [
    {
      "tex": "v_j=(\\cos(2\\pi j/3),\\sin(2\\pi j/3)),\\quad j=0,1,2",
      "explanation": "三个单位方向位于二维平面，完全构造，用于说明叠加几何。"
    },
    {
      "tex": "G_{ij}=v_i^\\top v_j,\\qquad G_{ij}=-1/2\\ (i\\ne j)",
      "explanation": "非对角项不为零，说明线性读取可能相互干扰，不是任何真实模型的相关矩阵。"
    },
    {
      "tex": "\\Delta f=f(\\operatorname{intervene}(h))-f(h)",
      "explanation": "干预效应必须说明替换规则、目标输出和对照；非零变化本身不足以证明语义命名。"
    }
  ],
  "table": {
    "headers": [
      "方法或判断",
      "控制变量与测量",
      "失效条件及解释"
    ],
    "rows": [
      [
        "激活观察",
        "独立输入上检验相关与反例",
        "可读取不等于被模型使用"
      ],
      [
        "注意力图",
        "连同value、残差和下游分析",
        "权重不是最终贡献"
      ],
      [
        "消融/补丁",
        "多基线、随机方向及副作用控制",
        "离分布干预可能误导"
      ],
      [
        "机制预测",
        "预先预测行为变化并复核",
        "解释流畅不等于忠实"
      ]
    ]
  },
  "protocol": [
    "固定模型哈希、层、激活位置和任务，划分发现/验证输入。",
    "保存特征提取与命名规则，加入相似概念及反例。",
    "同时检查相关、必要性和充分性，避免只做一种干预。",
    "使用匹配幅度随机方向及多个替换基线，测非目标副作用。",
    "在新提示、语种和任务变体上复验，报告解释失败比例。",
    "预先登记机制预测并公开代码，区分玩具几何与真实回路证据。"
  ],
  "diagram": [
    "固定任务与内部记录",
    "候选方向及特征命名",
    "独立相关性检验",
    "消融 / 补丁 / 随机对照",
    "必要性与充分性",
    "新输入上的机制预测"
  ],
  "analytic": "superposition-geometry",
  "limitations": [
    "二维构造不是已发现模型回路。",
    "探针读取不证明下游使用。",
    "干预可能产生离分布状态。",
    "机制结论依赖模型与任务。"
  ],
  "keywords": [
    "interpretability",
    "superposition",
    "interventions"
  ],
  "peer_reviewed": false,
  "journal_publication": false,
  "disclosure": {
    "authorship": "AI辅助研究综述；Codex（AI）依据所列一手文献撰写并进行解析推导。",
    "figures": "概念封面与科学图解分开；科学图由公开公式和明示假设计算，原始论文结果均注明归属。",
    "review": "本站没有本篇独立同行评审或原创模型实验记录。"
  },
  "section_kinds": [
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review",
    "source_based_review"
  ],
  "sources": [
    {
      "id": "arxiv-2209.10652",
      "title": "Toy Models of Superposition",
      "authors": ",  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,  ,",
      "url": "https://arxiv.org/abs/2209.10652",
      "date": "2022-09-21",
      "accessed_at": "2026-10-21",
      "locator": "Abstract; 原文方法、理论和实验章节见作者原始论文链接",
      "note": "一手论文；原文研究归属原作者。本文的推导示例与建议实验方案另行标注，不冒充原论文或本站实测。"
    },
    {
      "id": "arxiv-1902.10186",
      "title": "Attention is not Explanation",
      "authors": ",",
      "url": "https://arxiv.org/abs/1902.10186",
      "date": "2019-02-26",
      "accessed_at": "2026-10-21",
      "locator": "Abstract; 原文方法、理论和实验章节见作者原始论文链接",
      "note": "一手论文；原文研究归属原作者。本文的推导示例与建议实验方案另行标注，不冒充原论文或本站实测。"
    },
    {
      "id": "attention",
      "title": "Attention Is All You Need",
      "authors": "Vaswani et al.",
      "date": "2017-06-12",
      "url": "https://arxiv.org/html/1706.03762v7",
      "locator": "v7; sections 3-4; Table 2 (EN-DE only)",
      "note": "Transformer uses scaled dot-product attention; full attention has quadratic pair complexity. EN-DE big model 28.4 BLEU; no cross-version EN-FR claim.",
      "accessed_at": "2026-10-15",
      "rights": "Original source remains under its own license; only attributed factual synthesis, no source figures or long excerpts."
    }
  ],
  "figures": {
    "mechanism": {
      "url": "/guides/assets/v5-mechanism-mechanistic-interpretability-4a354765fe761ca5.png",
      "kind": "conceptual_method_flow",
      "data": [
        "固定任务与内部记录",
        "候选方向及特征命名",
        "独立相关性检验",
        "消融 / 补丁 / 随机对照",
        "必要性与充分性",
        "新输入上的机制预测"
      ]
    },
    "analysis": {
      "url": "/guides/assets/v5-analysis-mechanistic-interpretability-3f0b3a2426a127dc.png",
      "kind": "analytical_or_constructed_example",
      "data_url": "/guides/assets/v5-data-mechanistic-interpretability-fd551b5a16317c41.json",
      "assumptions": "Three unit vectors at0,120,240 degrees; no trained network or discovered feature."
    }
  },
  "cover_provenance_url": "/guides/assets/v5-provenance-mechanistic-interpretability-017b3497f883c22c.json",
  "source_revision": "ed6feca8fdc75c8de482fd69dedf17cbc746a9d1"
}