1. That suffocating OOM moment↗
Imagine three in the morning. The terminal flashes the familiar error again: “CUDA out of memory”. Loading a 70B model at the chosen precision and configuration has failed. The fan is still running; social media is already full of the technology giants’ next generation of “monsters”.
An anxiety follows: in a race measured in compute, must an individual researcher without those resources remain outside?
Scarcity of compute is hardly a gift. It prevents some experiments and prolongs others. It also makes a question concrete: if the system cannot keep growing, where else might understanding come from?
The “aesthetics of scarcity” here means finding a scale at which the tools at hand can still touch a worthwhile question, without romanticising the difficulty itself.
2. Reconstructing the object: the alchemy of language↗
Treat a model solely as a product of parameter scale, and research seems obliged to follow “larger”. Seen differently, it is also an existing system that remains incompletely understood. The question can shift from making it to organising the computation it performs next.
Chain-of-thought prompting offers an example, but two frequently conflated studies should be separated. Wei and colleagues’ CoT work uses few-shot demonstrations containing reasoning steps. The zero-shot instruction “Let’s think step by step” comes from Kojima and colleagues. Both improved performance in the models and tasks studied, rather than awakening universal reasoning with a sentence.
The image of an incantation is appealing. Research has to take it apart: what do the examples supply, what does extra generation change, and does the benefit survive beyond the original kinds of problem? Tree of Thoughts extends the inquiry to candidate paths, evaluation and search. Borrowing the intuition of branching thought does not establish that the brain executes the same algorithm.
These methods can leave model weights unchanged without being computationally free. More paths require more generation and evaluation. Even the alchemy of language needs an account of time, calls and criteria for success.
3. Folding space: making room in a snail shell↗
With one consumer graphics card, device memory resembles a room of fixed dimensions. If extension is unavailable, the question becomes how its interior can be rearranged.
Adapting an existing model to specialised legal text, for example, need not update every weight. LoRA freezes the base and trains low-rank updates to selected matrices. Absorbing the conventional scaling factor into the factors gives:
Here , and , with smaller than both dimensions. The second line counts only the trainable scalars in these two matrices.
For a matrix with , the factors contain 8,192 trainable scalars instead of updating 262,144 original entries. The base must still be stored and forward computation still performed. Fewer trainable parameters do not shrink the whole training process in the same proportion. The previous essay’s estimate distinguishes weights from runtime memory.
The craft of folding space lies in specifying what is discarded and what is preserved. More aggressive quantisation and adaptation to specialised tasks are research questions; data and tests must establish retained ability and a fair comparison. A small model beating a larger one on one score cannot alone establish overall capability or application safety.
4. Tracing knowledge to its source: between data and truth↗
Instead of endlessly enlarging the search through a noisy ocean of data, one can return to the source and polish a piece of jade. That is the appeal of data work.
Textbooks Are All You Need trains the code model phi-1 using selected web data and synthetic textbooks and exercises. This 2023 example makes the organisation of training material a serious research subject. It does not establish that intelligence comes only from data, independently of architecture or optimisation. Its training still required compute.
Curating a dataset for a smaller language or specialised domain can resemble a historian’s work: identifying sources, interpreting terms and recording disagreements. Manual work does not automatically mean quality, however, and no dataset is flawless. Whose expressions are included, whose context is removed and how labels are settled all change the world presented to the model.
“Data as truth” is therefore better treated as a question to examine carefully. A dataset sets operational conventions for sampling and evaluation; it does not thereby acquire authority to define truth. Its research value must be established through coverage, errors, leakage checks and relevant baselines, rather than guaranteed by the curator’s effort.
5. Weaving time: a system rather than an island↗
Look further out, and a model need not be an island. It can retrieve literature, use tools and encounter external information over several steps. The early RAG work combines parametric knowledge with external retrieval; later workflows called RAG are not all identical to that original, fine-tuned model.
How should those steps be arranged? Are the retrieved materials current? What happens when sources conflict? Do citations actually support the conclusions? Without supporting evidence, does the system stop or fill the gap with fluent language?
These questions can be studied in workflows without retraining the base model. Inference, retrieval, testing and maintenance still have costs. Adding retrieval and self-checking steps does not automatically add reliability. If successive steps repeat the same error, the workflow merely weaves it more tightly.
A more complicated agent does not acquire greater scientific value by accumulating functions. It needs a specific failure mode to address, a comparison with simpler workflows and a reason the extra time is worthwhile.
In the shadow of giants, it remains possible to study how language organises computation, how data carries knowledge and how systems interact with the world. These questions do not oppose research at scale, or become superior through scarcity. They offer another way to build: where expansion cannot be unlimited, examine the connections more closely.