待翻譯：The Sequence Knowlege #907: The Brain Transplant: Distilling Transformers Into Other Architectures

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

Every form of distillation in this series so far has quietly preserved one thing: teacher and student spoke the same dialect. A small transformer learned from a big transformer. The student was a compressed copy, then a more capable apprentice, then a reasoner trained on traces — but underneath, it was always the same kind of machine, attention layers stacked on attention layers, differing only in size. Cross-architecture distillation breaks the last shared assumption. Here the teacher is a transformer and the student is not — it’s a state-space model, or a linear RNN, or some gated recurrent thing that has never computed an attention matrix in its life. You take a fully trained transformer and pour its capability into a fundamentally different computational substrate, and somehow the capability survives the transplant. The first time you see it work, it feels a little illicit, like recovering a person’s memories after swapping out their brain for different hardware. This is the strangest corner of the distillation world, and also one of the most economically loaded. So it’s worth understanding why anyone would attempt something this perverse — and why, against reasonable expectations, it works. The Arbitrage Read more