Imagine understanding a model as sculpture: a structure seems hidden in the stone, and the chisel is meant to reveal it.
If the previous essay asked whether the ruler was bent, this one looks inside the marble. Behavioural curves provide clues. Can internal computation provide a different kind of evidence?
It is too early to announce that the statue has been found. Seeing part of a structure and anticipating what the whole stone will become remain different achievements.
Chapter 1: Induction heads, a clue inside the box↗
In 2022, Olsson and colleagues’ In-context Learning and Induction Heads examined the relationship between induction heads and in-context learning.
An induction head exhibits a simple, useful pattern: find a similar earlier context and tend to continue with the token that followed it. The characteristic example is:
This supplies a tractable implementation of context-dependent prediction. The copying pattern is not a complete solution to every few-shot task, nor should every occurrence of A be followed by B.
Induction heads formed around the training stage when the in-context learning measure improved sharply, with a visible turning point in training loss. An important detail is what the measure means: better prediction at later than at earlier token positions. That is not identical to the claim that a few examples enable an arbitrary new task.
The evidence also differs by model. For small attention-only models, ablations and related analyses supplied strong causal evidence. For larger models containing MLPs, the evidence was primarily correlational. The former conclusion cannot simply be transferred to every large model.
This goes beyond observing an output curve: an internal computation is linked more precisely to a measured behaviour. Calling it the first mechanical explanation of all emergence would go much further. A sharp change is not automatically a physical phase transition, either.
Chapter 2: Grokking, generalisation arriving late↗
Grokking offers another counterintuitive story.
Power and colleagues’ 2022 study observed models on small algorithmic datasets that fitted the training data well while test performance remained near chance. Generalisation improved much later, after continued optimisation.
This does not mean every overfitted model eventually recovers. Task, data fraction, and optimisation choices affect whether it happens. It challenges a simpler reading: that once training performance is good and test performance poor, nothing worth waiting for can still be changing inside.
Nanda and colleagues’ mechanistic analysis studied small transformers performing modular addition: given , predict , with in the main analysis. The models used a sparse set of Fourier frequencies to map numbers onto circles and combine trigonometric features representing their sum.
One relation makes part of this understandable. With and , where indexes a frequency:
Products can therefore help represent addition. The model must also combine its frequencies and read out candidate answers. This identity alone is neither the complete circuit nor a proof of a modular-multiplication solution.
The authors constructed restricted and excluded losses by retaining or removing key frequencies. These measures revealed internal progress before test accuracy rose sharply.
They distinguished three phases in the main training run:
| Phase | Epochs in that run | Mechanistic interpretation |
|---|---|---|
| Memory | 0–1400 | Training performance improves before the final Fourier structure plays a clear role |
| Circuit formation | 1400–9400 | Generalising structure develops while the ordinary test curve remains uninformative |
| Cleanup | 9400–14000 | Memorising components diminish, a more compact generalising solution dominates, and test performance improves |
These times are not a universal schedule. The paper also shows changes across seeds and data fractions. Weight decay matters to this explanation; it is not the sole cause of generalisation everywhere.
An abrupt improvement in the visible score can follow a much more gradual formation of internal structure.
Think of a shadow: a movement outside need not mark the moment when the stone first began to change. But this explains studied tasks, rather than settling every case of emergence.
Chapter 3: Can the next turning point be predicted?↗
Explaining a process does not automatically deliver a prediction. Prediction first needs a target: which task, model family, metric, and future range?
3.1 First clue: loss and performance↗
Du and colleagues’ 2024 study found that pre-training loss helped organise downstream scores under fixed training conditions. An observed task threshold can become a predictive clue within that setup.
It cannot be transferred unconditionally across corpora, tokenisers, or architectures. Knowing one historical threshold does not reveal thresholds for every new ability.
3.2 Second clue: separate difficulty levels↗
Wu and Lo’s 2024 preprint grouped questions by average performance. On several multiple-choice tasks, they found U-shaped trends for harder questions and inverted-U trends followed by improvement for easier ones, proposing Slice-and-Sandwich as a prediction method.
This offers an account and predictive procedure for studied tasks and metrics, rather than reducing every complex task to two superimposed curves. Group definitions, which model observations are available when making a prediction, and performance on unobserved larger models all affect the clue’s value.
3.3 Third clue: skills as basis functions↗
Nam and colleagues’ NeurIPS 2024 paper represented skills as basis functions in an analytically tractable multilinear model, comparing its dynamics with networks trained on multitask sparse parity.
In this controlled setting, calibrating one parameter on the first skill predicted the order and curves of subsequent skills. That is meaningful progress under specific assumptions: task structure, skill definitions, and data-frequency distributions are supplied. It does not predict which unknown abilities an open-world LLM will discover.
A different paper, Predicting Emergent Abilities with Infinite Resolution Evaluation, is by Hu and colleagues at ICLR 2024. Its PassUntil method increases evaluation resolution through extensive sampling. It is a separate research route, rather than Nam’s skill model.
These studies make some defined prediction problems more tractable. Predicting future performance on a defined task and identifying an ability that has not yet been defined are still different targets. The latter is harder to test; that is not a proof of permanent unpredictability.
Chapter 4: A more restrained epistemology↗
What, then, has emergence changed?
It has not overturned a supposedly linear engineering worldview. Engineering already deals with thresholds, feedback, and nonlinear behaviour. The more specific lesson is that curves at different levels can move out of step: a training objective, a task score, and an internal structure can provide different information at different times.
Evolution, social change, and language development may also suggest slow accumulation followed by sudden visibility. Similar appearances do not establish common mechanisms, thresholds, or irreversibility. The experimental accessibility of AI lets some hypotheses be tested more closely. It does not turn a toy model into a law of existence.
Restraint need not diminish the excitement. Turning a grand word into an explanation that can be tested—and can fail—is where a field starts to gain explanatory force.
Chapter 5: The boundary of understanding↗
Part of the stone has been opened.
Induction heads, Fourier representations, and gradually forming structures behind delayed generalisation are findings from particular studies. They supply internal accounts of some behavioural changes. They do not make all emergence an instance of the same circuit reorganisation.
Understanding a mechanism can help prediction; it does not automatically justify predictions beyond the evidence.
The weather analogy should stop there too. Understanding dynamics still leaves a need for initial conditions, information, and computation. It does not prove that language models must be unpredictable in the manner of a particular chaotic system.
Scaling requires acknowledging what remains unexplained and actively seeking measurements and experiments that reduce uncertainty. Awe alone cannot answer what the next tenfold increase will bring. A small-model explanation cannot serve as a guarantee, either.
Perhaps the more honest stance for a creator is neither to pretend the answer is already known nor to turn the unknown into a mystery to revere. What remains in the stone must be distinguished, one cut at a time.