翻訳待ち：The Sequence Knowlege #907: The Brain Transplant: Distilling Transformers Into Other Architectures

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

Every form of distillation in this series so far has quietly preserved one thing: teacher and student spoke the same dialect. A small transformer learned from a big transformer. The student was a compressed copy, then a more capable apprentice, then a reasoner trained on traces — but underneath, it was always the same kind of machine, attention layers stacked on attention layers, differing only in size. Cross-architecture distillation breaks the last shared assumption. Here the teacher is a transformer and the student is not — it’s a state-space model, or a linear RNN, or some gated recurrent thing that has never computed an attention matrix in its life. You take a fully trained transformer and pour its capability into a fundamentally different computational substrate, and somehow the capability survives the transplant. The first time you see it work, it feels a little illicit, like recovering a person’s memories after swapping out their brain for different hardware. This is the strangest corner of the distillation world, and also one of the most economically loaded. So it’s worth understanding why anyone would attempt something this perverse — and why, against reasonable expectations, it works. The Arbitrage Read more