The Sequence Knowledge #898: The Trace Is the Teacher: Distilling Reasoning Into Small Models
From the release of DeepSeek R1, distillation in reasoning models have become one of the most common techniques in frontier AI.
In January 2025, DeepSeek took its big reasoning model, R1, and used it to generate around 800,000 worked solutions — long, rambling chains of thought, complete with false starts, self-corrections, and the occasional “wait, let me reconsider.” They filtered these traces for correctness and readability, and then did the most boring thing imaginable with them: plain supervised fine-tuning on a handful of off-the-shelf open models — Qwen at 1.5B, 7B, 14B, 32B; Llama at 8B and 70B. No reinforcement learning. No reverse KL. No on-policy sampling. No teacher-as-critic. Just next-token prediction on the teacher’s transcripts.
And it worked spectacularly. The distilled 32B model started solving competition math it had no business solving. The 7B model began verifying its own work and branching its reasoning mid-stream — emergent behaviors nobody trained into it directly. A grab-bag of small dense models suddenly reasoned like something ten times their size.
If you internalized the arc of this series — the painstaking migration from forward to reverse KL, the whole argument that you cannot just imitate a teacher’s trajectories because the student will never be on them at inference — your first reaction should be mild outrage. We spent an entire installment establishing why naive sequence-level imitation is the wrong tool, and then the single most important reasoning-distillation result of the decade is naive sequence-level imitation. What gives?
The answer is the whole story of this installment, and it turns out to be more interesting than either “imitation works” or “imitation doesn’t.”
The Trace, Not the Answer
Read more