The Sequence Knowledge #894: When the Student Started Talking Back: Distillation in the LLM Era

Looking back at the 2015 distillation paper, what’s striking isn’t the temperature trick or the dark-knowledge framing --- it’s the world the paper quietly assumed. There was a fixed input distribution. There was a teacher that produced a probability vector over a closed set of classes. There was a student trained to match that vector. Run the dataset through both, compute the loss, backprop. The pipeline had a kind of mechanical innocence to it. Everything stayed in its lane.

Then language models arrived, and one by one, every assumption broke.

This installment is the story of that breaking. It is the part of the distillation arc where the field stopped thinking about compression --- making a smaller copy of a fixed function --- and started thinking about capability transfer: getting a small model to actually do something hard, with a larger model’s help. The shift took about five years, and it went through three recognizable stages, each of which looked at the time like a minor engineering improvement and turned out, in hindsight, to have moved the conceptual furniture.

Stage One: Sequences Are Not Pictures

Read more