The Sequence Knowledge - Issue 916: From Thinking Longer to Learning Better

There’s a scaling law hiding in your inference bill. The great discovery of the reasoning-model era was that you could buy intelligence at test time. Let the model think longer — generate a chain of thought, sample sixteen candidates and vote, run a tree search over reasoning paths, draft and self-verify — and accuracy climbs, often dramatically, without touching a single weight. Test-time compute became the third axis of scaling, after parameters and data. Every frontier lab reoriented around it. But there’s something conceptually odd about paying for the same cognition over and over. If your model needs to sample sixteen candidates and majority-vote to reliably answer a class of question, then in some sense the ensemble of sixteen samples plus the vote is the real model — a better model that happens to be implemented as an expensive inference-time ritual. And the moment you phrase it that way, a distillation-shaped question appears: can you take that better model and compress it back into the weights? Can you train the network to produce, in one forward pass, what the ritual produces in sixteen? This is test-time compute distillation, and it’s the strangest teacher this series has met yet. The teacher is not a bigger network. The teacher is the same network, given more time to think. You are distilling a model into itself. The Amortization Move Read more