The Sequence Knowledge - Issue 911: Distilling Diffusion and Multimodal Models

Text distillation teaches a smaller model to imitate an answer. Diffusion and multimodal distillation must compress trajectories, distributions, motion, and the semantic geometry between different worlds. Text distillation is relatively easy to narrate. A large language model sees a prompt and produces a distribution over the next token, or perhaps a complete response. A smaller model is trained to imitate that behavior. The teacher says “Paris”; the student learns to say “Paris.” The teacher writes a good explanation; the student learns the shape of the explanation. Diffusion distillation is stranger. A diffusion model does not emit an image in one clean forward pass. It starts from noise and repeatedly edits that noise until a coherent sample appears. Generation is a trajectory, not an answer. The model is less like a database query and more like a sculptor taking dozens of tiny cuts. Read more