2026-07-13 04:00 UTCOriginal source2 min readUpdated: 2026-07-13 08:10 UTC

A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions

This paper proposes a unified approach to explain the mechanism of knowledge distillation (KD) in large language models (LLMs). By decomposing the output into interactions, it reveals that KD commonly sparsifies interactions—student models retain fewer interactions for inference. Performance differences stem from handling complex interactions, leading to a novel Complex Interaction Penalty (CIP) loss that improves various KD methods consistently.

SourcearXiv Machine LearningAuthor: Qingzhuo Wang, Ruiyang Qin, Zhenxin Qin, Wen Shen, Zhihua Wei

Article intelligence

EngineersAdvanced

Key points

Explores the common mechanism of KD via interaction decomposition, finding interaction sparsification as universal. Student models keep fewer interactions, suppressing others to zero. Performance varies by ability to handle complex interactions; higher sparsity of complex interactions yields better performance.
Proposes a plug-and-play Complex Interaction Penalty (CIP) loss to enforce complex interaction sparsity during distillation, consistently boosting KD methods on both in-domain and out-of-distribution benchmarks.

Why it matters

This matters because explores the common mechanism of KD via interaction decomposition, finding interaction sparsification as universal. Student models keep fewer interactions, suppressing others to zero. Performance varies by ability to handle complex interactions; higher sparsity of complex interactions yields better performance.

Technical impact

May affect model selection, inference cost, product capability, and evaluation benchmarks.

This panel is AI-generated and reviewed for accuracy.

-->

[Submitted on 5 May 2026]

Title:A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions

View a PDF of the paper titled A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions, by Qingzhuo Wang and 4 other authors

View PDF HTML (experimental)

Abstract:Despite the success of knowledge distillation (KD) in Large Language Models (LLMs), the underlying mechanism behind its efficacy remains unclear. In this paper, we propose a unified approach to explore the common mechanism of various KD methods using interactions. Specifically, we decompose the output score of the LLM into the sum of numerous interactions. Each interaction represents a nonlinear relationship involving a set of input variables (e.g., words). Based on the decomposed interactions, we discover that the common mechanism underlying various KD methods is the sparsification of interactions, i.e., student models retain fewer interactions for inference while suppressing other interactions to zero effects. Furthermore, we discover that the performance variance across different KD methods arises from their capabilities in handling complex interactions. A KD method typically yields better performance if it enables the student model to achieve higher sparsity of complex interactions. Motivated by these insights, we propose a plug-and-play loss function called Complex Interaction Penalty (CIP) to explicitly enforce the sparsity of complex interactions during the distillation process. Extensive experiments demonstrate that integrating CIP consistently improves the performance of diverse KD methods on both in-domain and out-of-distribution benchmarks.

Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Science and Game Theory (cs.GT)

Cite as: arXiv:2607.08776 [cs.LG]

(or arXiv:2607.08776v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2607.08776

arXiv-issued DOI via DataCite

Submission history

From: Wen Shen [view email] [v1] Tue, 5 May 2026 07:24:01 UTC (8,517 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions, by Qingzhuo Wang and 4 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.LG

new | recent | 2026-07

Change to browse by:

cs cs.AI cs.CL cs.GT

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)