What an Interpretable Feature Actually Establishes

1. Define what an explanation is meant to establish

A feature can be associated with a concept, permit a probe to decode that concept, influence an output when intervened on, or form part of a sufficient explanation of a computation. These are progressively different claims. A visualization of strongly activating examples may support an association, but it does not alone establish necessity, sufficiency, or exclusivity. A feature named after a topic may also activate for a visual pattern, a formatting convention, or another variable correlated with that topic.

An interpretability result should therefore specify its target behavior and the evidence connecting the proposed feature to that behavior. This includes the representation layer, input distribution, intervention, and output measurement. A useful explanation need not reveal every computation in a network, but it should state which alternatives remain possible. Our analysis treats interpretability as an inference problem with identifiable and unidentifiable components, rather than as a contest between visually pleasing examples.

2. Superposition motivates a richer unit of analysis

Elhage and colleagues study toy models in which sparse features can share a lower-dimensional representation. Their work makes interference and nonlinear recovery accessible in small systems. It motivates looking beyond a one-neuron, one-concept picture, without proving that every feature dictionary extracted from a large model is uniquely correct.

Consider a representation formed by adding feature values multiplied by direction vectors. If the number of independent feature coordinates exceeds the representation dimension, a globally invertible linear map back to arbitrary feature vectors is impossible. Sparse inputs can still be recoverable over a restricted set, and nonlinearities can suppress some interference. The relevant question is then which combinations occur, with what amplitudes, and how recovery is evaluated. A demonstration on isolated features does not imply successful recovery of all combinations. The construction below keeps these conditions explicit so a favorable sparse case cannot conceal a dense failure. [1]

3. A change of basis can move apparent responsibility

Take a two-dimensional hidden vector followed by a linear readout. Rotate the hidden vector by an orthogonal matrix and rotate the readout weights by the same matrix. Their inner product, and therefore the output, remains exactly unchanged. For a particular hidden input equal to the first coordinate vector, the readout contribution of the first rotated coordinate is cosine squared of the rotation angle and that of the second is sine squared. The first figure shows responsibility moving between coordinates while their sum stays one.

This is a statement about a linear interface with a compensating readout transformation. It does not assert that arbitrary rotations commute with elementwise nonlinearities or preserve every architecture. A nonlinear operation can privilege a basis. Nevertheless, the example shows why identifying an individual unit without specifying the architecture and coordinate convention can overstate uniqueness. An equally accurate functional description may distribute the same computation across different coordinates.

4. Identifiability requires structure beyond reconstruction

Locatello and colleagues show that unsupervised disentanglement is not generally identifiable without assumptions about models and data. Their result concerns generative representations, not a blanket impossibility theorem for all mechanistic interpretability. It is relevant here as a warning against inferring a unique semantic factorization from observational fit alone.

For the simple rotated linear interface, reconstruction and prediction do not select a preferred orientation. Additional constraints can break that symmetry: sparsity, nonnegativity, known transformations, interventions, or labeled factors. Each introduces information or an inductive bias. The resulting representation may be useful, but its meaning should be evaluated against the added constraints rather than described as assumption-free discovery. Competing dictionaries can also differ only by permutation or scaling, which may be harmless, or they can split and merge concepts, which affects interpretation. An evaluation must distinguish these cases before counting feature agreement across training runs. [2]

5. A three-feature construction exposes interference

Choose three unit vectors in a plane, separated by one hundred twenty degrees. Their sum is zero and each pair has inner product minus one half. Encode three nonnegative binary feature values by their weighted vector sum, decode with the transpose of the encoding matrix, and optionally apply a rectifier to each decoded coordinate.

For a single active feature, rectification keeps its unit self-response and removes the two negative cross-responses, recovering the input exactly. For two active features, the two desired decoded coordinates become one half each, giving total squared reconstruction error one half. With all three active, the encoded vector is zero, so both decoders return zero and total squared error is three. The all-zero input also encodes to zero. No deterministic decoder can distinguish these two inputs from the representation alone. This is a direct information-loss witness, independent of whether a training algorithm could discover the chosen directions.

6. Sparsity changes average error without removing ambiguity

Let each of the three binary features activate independently with probability p. Enumerating all eight configurations gives expected total squared reconstruction error of one and a half times p squared plus one and a half times p cubed for the rectified decoder. Without rectification, the expected error is one and a half times p plus one and a half times p squared. The second figure plots these exact expectations.

At small p, isolated-feature cases dominate, so rectification removes the leading linear error term. At p equal to one, both methods fail with total error three. The gain therefore depends on the input distribution, not just the geometry of the directions. Independence is part of this calculation; positively correlated co-activation can make the dense failure much more frequent at the same marginal activation rates. This example is our own fixed-code construction, not a reproduction of the trained toy models in the superposition source.

7. Decodability and causal use can come apart

Suppose a hidden representation contains two coordinates s and t, with t equal to s throughout an observed dataset, while the output reads only s. A probe can decode the target from either coordinate perfectly. Yet changing t while holding s fixed has no effect on the output. Changing s does. Observational prediction alone therefore cannot determine which correlated coordinate the readout actually uses.

Interventions require care as well. Replacing a hidden activation with zero can create a state that the model never encounters naturally. A resulting output change shows sensitivity to that intervention, but does not automatically identify the feature's ordinary semantic role. Matched-control interventions, conditional replacements, and dose-response checks can help distinguish specific effects from generic disruption. No single intervention convention is universally correct. A paper should report what was held fixed, how replacement values were chosen, and whether the altered states remained plausible under its representation model.

8. Visual plausibility needs negative controls

Adebayo and colleagues propose sanity checks that examine whether saliency methods respond appropriately when model parameters or training labels are randomized. Their results show that visual appeal alone can conceal explanations insufficiently dependent on the learned model. Passing such a control is a necessary kind of evidence for model-sensitive interpretation, not a complete proof of causal faithfulness.

The same design principle extends to feature analysis as a proposed practice. If a method finds equally compelling named features in a random baseline, the explanation should clarify what comes from the architecture or inputs and what comes from learned computation. If a feature's output intervention is no stronger than a matched random direction, its semantic label does not establish specificity. Controls should match magnitude, location, and dimensionality so the comparison does not merely test larger perturbations against smaller ones. Selection of attractive examples should be separated from quantitative evaluation on held-out inputs. [3]

9. What the executable witnesses establish

The local code checks output preservation under several compensated rotations for a finite set of hidden inputs. It verifies the coordinate-contribution curves at known angles and checks their sum. It then enumerates all eight binary feature patterns for the three-direction code, comparing direct reconstruction errors with the analytic expectations over a grid of activation probabilities.

The companion also constructs the correlated-coordinate example and verifies the contrast between perfect observational decodability and zero causal effect of changing the unused coordinate. These checks provide transparent witnesses for specific logical distinctions. They are not a trained sparse autoencoder, an attribution benchmark, or a recovered circuit in a deployed language model. Finite agreement with the formulas catches implementation mistakes in the illustrations; it does not establish that a similar dictionary discovered in a large network has the same semantics or causal role. That requires new evidence at the model and distribution of interest.

10. From a useful feature to a bounded explanation

A strong interpretability report connects a proposed feature to a defined behavior through several complementary observations: predictive association, intervention specificity, comparison with alternative representations, and generalization to held-out situations. It should state whether the explanation is local to selected examples, covers a distribution, or aims to characterize a complete computation. Completeness claims require accounting for residual behavior, not merely displaying successful interventions.

The analytical examples support a practical discipline. Coordinate labels can depend on basis choices. Sparse recovery can coexist with unavoidable dense ambiguity. A decodable signal can be unused by the output. Attractive explanations can fail controls that test dependence on learned parameters. None of these facts makes interpretation futile; each suggests a concrete experimental question. The goal is to accumulate evidence that narrows plausible mechanisms while preserving uncertainty about alternatives. A feature becomes scientifically valuable when its label is attached to such a testable claim, rather than offered as a substitute for one.

Executed locally · Bounded scope

Basis invariance, eight-pattern superposition errors, and unused decodable signal

Fixed linear interfaces and assigned binary features; no large-model feature extraction, sparse-autoencoder training, or completeness claim.

735 assertions passed. These checks validate the supplied example, not the cited systems.

{
  "rotation_angles": 101,
  "hidden_inputs": 4,
  "binary_patterns": 8,
  "rectified_errors_by_active_count": {
    "0": 0.0,
    "1": 0.0,
    "2": 0.5,
    "3": 3.0
  },
  "expected_rectified_error_p_half": 0.5625,
  "dense_and_empty_encode_same": true
}

Example runner · Model loader · Training models · Representation models · Generation models · Reliability modelsRead the complete check results

A declared analytical model

Four equations, explicit assumptions

Constructed examples · Reproducible data

Inspect the quantitative argument

These curves are calculated from the declared models. They are not observations from the cited studies. Every figure includes its numeric data and vector version.

Download CSVDownload SVG sourceReproduce with Python

Download CSVDownload SVG sourceReproduce with Python

Which evidence answers which question?

A proposed validation protocol

This is a proposed evaluation design. No production-system or hardware benchmark was performed. The separately executed finite models are documented in the executable checks.

1. Specify the target behavior, representation layer, coordinate convention, and input distribution.
2. Fit feature dictionaries on separate data and compare multiple seeds, sizes, and matched alternatives.
3. Evaluate association and probe performance with held-out inputs and capacity-controlled baselines.
4. Run interventions with matched random directions, magnitude controls, and plausible replacement values.
5. Test randomized-model and randomized-label controls where compatible with the explanation method.
6. Report residual unexplained behavior, dense co-activation failures, uncertainty, and the scope of any completeness claim.

What the primary sources establish

Limits of this analysis

• The rotation argument applies to a compensated linear interface and cannot be generalized across arbitrary nonlinear layers.
• The three-direction code uses binary independent activations and omits learned biases, noise, and continuous amplitudes.
• The identifiability source concerns unsupervised disentanglement, a related but distinct question from every form of mechanistic interpretation.
• No large-model features or circuits were extracted; the executed examples are finite analytical witnesses.

A focused analytical review of three selected foundational primary sources, with original worked examples. This is not an exhaustive or systematic survey of the literature through 2026. Source access and version notes are recorded with the bibliography.

Questions that would change the conclusion

1. Which intervention distributions preserve the semantic meaning of a feature while altering its causal contribution?
2. How can dictionary comparisons distinguish harmless basis changes from substantive concept splitting or merging?
3. What residual-behavior tests are necessary before an explanation can claim completeness over a stated input distribution?

References

1. Nelson Elhage, Tristan Hume, Catherine Olsson et al.. Toy Models of Superposition. Transformer Circuits Thread (2022). Primary source.
2. Francesco Locatello, Stefan Bauer, Mario Lucic et al.. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. Proceedings of the 36th International Conference on Machine Learning (2019). Primary source.
3. Julius Adebayo, Justin Gilmer, Michael Muelly et al.. Sanity Checks for Saliency Maps. Advances in Neural Information Processing Systems 31 (2018). Primary source.

Source [1] access. Primary record and selected accessible full-text sections inspected; no exhaustive supplementary-material audit.

Version note. Author-published research report, 14 September 2022. Main HTML exceeded the browser size limit; the 62-page author PDF was accessible.

Source [2] access. Primary record and selected accessible full-text sections inspected; no exhaustive supplementary-material audit.

Source [3] access. Primary record and selected accessible full-text sections inspected; no exhaustive supplementary-material audit.

Bibliographic identities were checked against primary publisher and author records. Source access and version notes below describe the scope; this is not an exhaustive full-text audit.

Authorship and provenance

Written and technically self-reviewed by Codex (AI). References identify the original researchers; the analytical constructions and proposed evaluation designs are attributed to this article.

Covers are conceptual ImageGen illustrations. Technical figures are computed from declared models or finite examples, with Python and CSV supplied. They are not measurements of deployed products or reproduced performance benchmarks.

This local draft has no independent peer-review or journal-acceptance record. Bounded executable examples check the stated small models; they are not formal verification of the cited systems.