1. A man who seemed to live only in today↗
In 1953, 27-year-old Henry Molaison underwent an operation that changed the rest of his life. To relieve severe epilepsy, the neurosurgeon William Scoville removed tissue from both medial temporal lobes, including hippocampal and neighbouring regions.
The seizures became less disabling, but continued. A profound memory impairment followed: recent events rarely became lasting new memories. At an examination in 1955, he could not recall a conversation he had just had outside the room. Earlier memories and general reasoning were relatively preserved, with some loss extending into the years before surgery. This is H.M. in Scoville and Milner’s original case report, rather than someone whose entire past remained intact and whose capacity for new memory was literally zero.
“Living only in today” is a metaphor for that predicament. Its unsettling quality is that a person can still talk with you, while the conversation may never reach his tomorrow. Memory cannot be understood simply as a safe containing all the past; different functions and injuries can leave very different absences.
2. Fast-forward seventy years↗
In 2022, Kevin Meng and colleagues posed another question about memory: could a factual association expressed by a language model be changed without retraining the whole model?
For example, could the model be made to associate the Eiffel Tower with Rome instead of Paris? The intervention changes its answer, of course, rather than the geography.
This shares an unsettling question with H.M.’s operation: what, exactly, changes when memory is altered? It does not share a mechanism. Human amnesia, a model’s factual associations, and a mathematical weight update do not belong on the same anatomical map merely because all three invite the word “memory”.
The NeurIPS 2022 paper was Locating and Editing Factual Associations in GPT, and its method was called ROME. Turning to a page and changing a word was no longer just an image. Whether the other pages remained intact was a separate question.
Does knowing where to look mean knowing where to edit?
3. Knowledge neurons: coordinates of memory?↗
In Knowledge Neurons in Pretrained Transformers, Dai and colleagues studied factual cloze tasks in BERT. Attribution helped identify feed-forward neurons whose suppression or amplification affected the expression of relevant answers.
The library metaphor still helps: some shelves matter for retrieving particular knowledge. That does not give each fact a private compartment. An intervention’s effect at one location does not establish that the fact exists only there.
The AAAI 2024 paper Journey to the Center of the Knowledge Neurons examined cross-language overlap in multilingual models and “degenerate knowledge neurons”. Degeneracy here concerns overlapping functions, rather than shallow learning: different neuron sets supporting a fact can sustain its expression when some members are suppressed.
Failing to find an irreplaceable neuron therefore does not imply that no relevant computation exists. Finding cross-language overlap does not give every language a shared, independent address either. Coordinates help locate memory; they need not exhaust it.
4. ROME: rewriting a page↗
An imagined subject–relation–object spreadsheet makes the editing target easy to grasp: replace the last item in “Eiffel Tower—located in—Paris”. There is no literal spreadsheet waiting inside the model.
4.1 From tracing to an update↗
ROME’s study first traces information by corrupting subject embeddings and restoring selected clean internal states to test answer recovery. The editing method then computes a target representation for a chosen MLP and makes a rank-one update to one weight matrix.
Its structure has this general form, shown schematically rather than as ROME’s complete solution:
For , the vectors have dimensions and . Their outer product can change many matrix entries. For another input , the output changes by ; other inputs may therefore be affected unless that inner product vanishes.
Low rank constrains the update’s structure. It does not automatically bound its size or guarantee local behaviour. Whether the finest scalpel touches only its target has to be established from what happens afterwards.
4.2 What changed, and what else was touched?↗
The original paper tests original prompts, paraphrases and neighbouring facts. These separate three questions about an edit:
| Check | Question | What it does not establish |
|---|---|---|
| Efficacy | Does the target prompt favour the new answer? | Every phrasing has changed |
| Paraphrase generalisation | Does the effect survive rewording? | It holds in every context |
| Locality | Are neighbouring or unrelated contents affected? | No side effects beyond the finite tests |
Even strong performance on all three does not identify a fact’s unique storage address. In Does Localization Inform Editing?, Hase and colleagues found that causal tracing did not predict the best MLP layer to edit in their main fact-overriding setting.
Knowing where something participates and knowing where best to operate are different questions.
Updating stale facts and reducing privacy leakage are worthwhile uses to explore. Changing a few answers, however, does not establish reliable erasure of private information. The same ability could implant false associations or biases. Greater power to edit brings a greater need to understand the edit’s reach.
5. Representation engineering: markings on a control panel↗
5.1 Moving along a direction↗
If knowledge editing resembles rewriting a page, representation engineering resembles searching for a control panel. In their 2023 preprint Representation Engineering, Zou and colleagues centre the reading and control of representations: derive directions from contrasting stimuli, then test whether interventions change the relevant behaviour.
Contrasting activations under different instructions can yield candidate directions; the paper also uses PCA to derive reading vectors from sets of differences. A simple additive intervention can be illustrated as:
Here : is an activation at a selected layer and position, a candidate unit direction, and a signed intervention strength. How the direction is estimated, where it is applied, and at what strength are part of the method.
An “honesty direction” names an experimental finding, rather than a context-free moral switch. Evaluation must distinguish better factual answers from a change in tone. A label on the panel cannot replace inspection of the wiring.
5.2 Refusal is not carved in stone↗
In the NeurIPS 2024 paper Refusal in Language Models Is Mediated by a Single Direction, Arditi and colleagues studied thirteen open chat models, up to 72 billion parameters. Interventions on one direction strongly affected refusal in the models and tests studied.
That is a warning: at least some safety-relevant behaviour depends on internal representations that can be changed. Yet refusal is only part of safety, and one direction is not a universal key. Reducing refusal does not remove every safety mechanism; increasing it does not guarantee safety.
Writing in sand remains a useful image. It directs attention to the material carrying a rule, without reducing an entire system to a line that can simply be wiped away.
6. SAEs and safety: an eraser that recognises only blue↗
The previous essay discussed SAEs as attempts to decompose activations into sparse features. A natural proposal follows: if some features participate in unwanted behaviour, could suppressing them reduce that behaviour?
The actual task matters. The 2024 preprint Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks evaluates SAEs through tasks including removing spurious classifier cues and perturbing probes. It does not establish complete deletion of dangerous knowledge.
Suppose a feature affects both harmful applications and ordinary chemistry. Suppressing it might reduce the former while damaging the latter; other pathways might preserve the original ability. These are risks to test, rather than an impossibility theorem about every SAE.
It is like trying to erase the sky from a painting, only to discover that the eraser recognises “blue”. The sea is blue too. So are the subject’s eyes.
The question is not merely whether the eraser is precise enough. It is whether “blue” is the right unit for this particular edit. Giving a feature a readable name does not draw a boundary around everything it affects.
7. The circuit-analysis problem: a suspect, or a different question?↗
In their 2023 Chinchilla circuit-analysis study, Lieberum and colleagues examined a limited capability in a 70-billion-parameter model: given the correct answer text in a multiple-choice question, how does the model produce the corresponding answer label?
They identified some attention heads and MLPs and investigated “correct letter” heads. Some subspace compression and feature explanations worked for ordinary answer labels. On a broader distribution including randomised labels, the explanation covered only part of the behaviour.
This relates to, but is distinct from, the GPT-2 backup circuits in the previous essay. The direct lesson here is that an explanation established on one task distribution needs testing on another.
The detective metaphor reaches a limit. Finding another route does not mean that a cunning suspect deliberately concealed a second skill. The researcher’s earlier questions may simply have left that route unexplored.
After “What did the analysis find?”, another question follows: what changes has this explanation survived?
8. Incompleteness: an epistemological ceiling?↗
Where does interpretability research end?
One hope is that improving tools will eventually make the components and computations explicable. A competing worry is that learned structures may resist compression into familiar human concepts.
Quantum mechanics offers a limited reminder. At the opening of his discussion of quantum behaviour, Feynman describes how atomic behaviour resists everyday pictures of waves or particles. Intuitive images can fail while abstract mathematical descriptions still advance understanding.
Neural networks may also demand new descriptive units. Geometry that is difficult to picture can still be calculated. Something that lacks a familiar name need not remain inexplicable forever.
“Incompleteness” is therefore a question here, rather than a demonstrated theorem. Neither quantum phenomena nor Gödel’s theorem establishes such a limit for neural networks. The useful question is what structure an explanation must preserve for prediction, intervention or auditing—and what it sacrifices to become easy to understand.
9. What, then, should be done?↗
The tension remains. There is real progress in tracing information, identifying features and circuits, and making targeted edits. Each advance also reveals gaps: feature absorption, unexamined pathways, collateral effects.
I favour a cautious realism.
First, interpretability is not a panacea. It is one part of a safety toolkit. Understanding a mechanism does not automatically resolve risks arising from deployment settings, access permissions or human use.
Second, partial understanding has value. Doctors do not need to understand every neuron; engineers do not track every atom. Their practices also rely on tested models, procedures and monitoring. The relevant question is whether the available understanding addresses a particular risk.
Third, I would like transparency to be the default. Access to internal states creates further opportunities for debugging and auditing. Visibility alone does not earn trust, though. What becomes visible must bear on the failures of concern and withstand testing.
Fourth, uncertainty has to be lived with. The possibility of incomplete understanding deserves preparation, rather than advance acceptance as fate. The practical question is which uses can responsibly proceed and which risks first require constraints.
10. The builders of Babel↗
In Genesis 11, people build a city and a tower, then cease building after their language is confused. The story admits many readings. For me, it leaves a question: can people build beyond their understanding?
Large language models make that question concrete. Designing an architecture, running its training and explaining every computation that emerges are different achievements. The tower can rise while its blueprint still contains blanks.
Interpretability tries to fill those blanks: tracing structures, checking foundations, asking where a failure might begin. Understanding the tower need not mean giving every brick a name.
For safety, the question I care more about is whether that understanding helps prevent the tower from falling on people. Control is one capability; prudence concerns how that capability is used when the evidence remains incomplete.
Viewed this way, interpretability brings philosophy and ethics back to the building site alongside technical work. Trying to understand what has been created also brings questions about explanation, responsibility and ourselves back into view.