We built a tower of Babel, only to find that we still cannot fully explain its echoes.
This does not mean every computation in a neural network is unknowable. In an open-weight model, the matrices, activation functions, and computation graph are available to inspect. The difficulty lies elsewhere: being able to execute the operations does not necessarily explain how they jointly produce an ability.
Think of a detective novel. The detective has diaries, movements, and testimony, but still lacks an explanation connecting the clues. “Why did he do it?” and “What processes made it happen?” sound close, yet call for somewhat different evidence.
The distance appears again with language models that write poems and code. Ask how a model did something, and its answer is still generated text, not a direct record of the computation. A plausible explanation begins as another clue.
1. Intelligence that was not written rule by rule↗
When a language model answers “2” to “1+1,” that does not mean it called a calculator's addition routine. It might have learned a generalisable computation, or relied on familiar expression patterns. The answer alone does not distinguish the routes.
Unlike a rule-based program, much of a neural language model's behaviour is acquired through parameter updates rather than specified individually by an engineer. That does not mean it never encountered the example: training text may certainly contain “1+1=2.” Learning and prior exposure are not separated by a clean boundary either.
The question becomes: what did it learn? Which representations and operations turn this question into this answer?
Mechanistic interpretability tries to turn such questions into testable accounts of computation. A neural network is not magic. But a definite process does not automatically yield a compact, reliable human understanding. Opening the weights does not, by itself, illuminate the forest.
2. Two ways of looking: telescope and microscope↗
Explanations can begin at different scales. A local explanation asks which inputs or internal processes affected this particular response. A global explanation asks what regularities or computations the model has learned across a set of situations.
The detective needs both “Why this street on Tuesday?” and “What explains this series of choices?” These perspectives can support each other. Local evidence can suggest a more general mechanism; a proposed mechanism must return to particular cases to be tested.
Scale makes that work harder. It does not follow that every attribution analysis costs nearly as much as retraining a model. Cost depends on the method, the number of interventions, and the access available. Sometimes the more basic difficulty is turning “understanding” into a sufficiently specific question.
3. The first tool: probing↗
3.1 What can be read from an activation?↗
Suppose a system's internal activity could be recorded while it processes language. Rather than searching for a single “Paris” or “capital of France” neuron, first ask whether a layer contains information that can distinguish the property of interest.
A probing classifier offers a direct procedure: freeze the original model, extract a layer's activations, and train a readout to predict a specified label, such as a token's part of speech. A linear classification probe can be written as:
Here is the representation at the selected position in layer , and is a property label with classes. The probe parameters are and . Only the readout is trained; the original model stays fixed. Accuracy should be evaluated on held-out data.
The approach is elegant, but the readout contributes to producing the answer. Belinkov's (2022) review makes the distinction explicit: extracting information from a representation does not show that the original model uses it for its own task.
Readable information is evidence of accessibility; using it to answer is a separate claim. Conversely, a failed probe shows that this readout and dataset did not succeed, not automatically that the information is absent. “Hume's Ghost” develops this distinction further.
3.2 Control tasks: how much did the probe learn?↗
Hewitt and Liang (2019) introduced control tasks that assign a fixed random label to each word type. Every occurrence of that word receives the same label, regardless of context. Success shows a probe's capacity to memorise word-to-output mappings without relying on the linguistic regularity under investigation.
Linguistic-task accuracy is therefore considered alongside control-task accuracy. Their difference is called selectivity. This is not merely shuffling every example's label and checking whether the training set can be fitted.
| Probe | POS accuracy (%) | Control accuracy (%) | Selectivity (pp) |
|---|---|---|---|
| Linear | 97.2 | 71.2 | 26.0 |
| MLP | 97.3 | 92.8 | 4.5 |
These default-setting results on ELMo's first layer show how nearly identical task accuracies can accompany very different memorisation capacities. Control tasks make the comparison more informative. They do not eliminate every alternative explanation or turn correlation into causation.
4. The second tool: the Logit Lens↗
4.1 Peeking into each chapter's notebook↗
In August 2020, nostalgebraist proposed a simple observation in “interpreting GPT: the logit lens”: if the output head converts a final hidden state into vocabulary scores, what happens when the same head reads an intermediate layer early?
For the common pre-LN architecture considered here, suppressing the position index and output bias, the readout is:
The unembedding maps into a vocabulary of size ; is the final normalisation. At a given position, is the next-token distribution read prematurely from layer . The linear map produces logits; softmax converts them into probabilities.
It is rather like peeking into the detective's notebook after every chapter. But “who the detective suspects” is a metaphor for a readout distribution, not a verified record of belief.
In the post's GPT-2 examples, some layerwise predictions gradually approach the final output. Others are hard to interpret early, or settle only late. “Refining a guess” is a useful perspective, rather than a smooth path guaranteed for every model and token. The post's addendum also revisits “input discarding”: failure to see information through this lens does not establish that the information has disappeared.
4.2 The Tuned Lens: translating between chapters↗
The Logit Lens borrows the final layer's readout, but earlier representations may not yet suit it. Belrose and colleagues' (2023) preprint learns an affine “translator” for each layer before applying the output head:
The trainable parameters are and ; the original model remains frozen. The readout is trained to approximate the model's final prediction distribution, rather than teaching each layer separate ground-truth answers.
Across the autoregressive models tested, this yielded more stable readouts. The authors also used feature-direction ablations to compare causal influences on the lens and the original model. It would therefore be wrong to say such work can only observe, never intervene. A better translator nevertheless does not recover every computation.
4.3 Does the model think in English?↗
Multilingual findings make the notebook metaphor particularly tempting. In controlled single-token completion tasks on Llama-2, Wendler and colleagues (ACL 2024) found that intermediate Logit Lens readouts could favour the English version of a semantically appropriate answer before shifting towards the target language in later layers.
This motivates studying English-biased representations. It does not establish that ChatGPT has an English inner monologue while answering in Chinese. The paper's proposed concept space retains information beyond the token readout; its authors also limit the conclusions to the model family and simplified tasks studied.
The language visible through a lens is not the whole language of a hidden state. English tokens being easier to decode and all information being translated into English are different explanations. This observation alone also cannot settle whether multilingual “understanding” is genuine.
5. From reading clues to testing them↗
There are now two kinds of tools: probes test whether specified information is readable, while the Logit Lens and Tuned Lens map intermediate states into comparable predictions. They open a window beyond inputs and outputs. A plotted trajectory does not, by itself, identify the causal process.
After reading a diary, the detective must still establish its relation to events. The next essay turns to features, circuits, and causal interventions: how does changing an internal variable affect the result, and how can that effect be distinguished from nonspecific damage or alternative pathways?
If a probe resembles a thermometer, activation patching is closer to a controlled operation. If the Logit Lens offers a glimpse of a notebook, a sparse autoencoder attempts another vocabulary for entangled representations. The metaphors point towards different kinds of work; they do not certify that the tools have uncovered the truth.
The unresolved question remains: when a model does not organise computation around the concepts people expected, how can an explanation stay close to its operation and remain open to human testing? Reading the black box may begin with recognising when the view belongs to the model, and when it belongs to the lens.