Skip to content
AI News HubLIVE
Original source2 min read

Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning

Summary

The paper introduces Activation-Prune-Merge (APM), a training-free framework for cross-scale heterogeneous model fusion. APM builds task-conditioned activation maps on a stronger donor model, prunes it to the recipient's architecture by selecting salient layers, hidden dimensions, attention heads, and MLP neurons, and injects the resulting slice with a micro interpolation coefficient. Across 16 benchmarks, APM raises the 3B recipient's average accuracy from 55.5% to 60.6%.

SourcearXiv Machine LearningAuthor: Jiahe Fan, Si Chen, Yinghao Hou, Aiyuan Zhang, Hong Xie
Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[Submitted on 31 Jul 2026]

Title:Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning

View a PDF of the paper titled Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning, by Jiahe Fan and 4 other authors

View PDF HTML (experimental)

Abstract:Heterogeneous model fusion seeks to combine models that differ in tasks, initializations, architectures, or scales. We study an underexplored cross-scale setting: improving a small recipient language model with a stronger donor despite substantial architectural mismatch. We ask whether useful capabilities can be transferred without explicit neuron-wise semantic alignment. Building on the observation that truncating a large model to a smaller architecture and injecting it with a tiny mixing weight can already improve the recipient, we propose Activation-Prune-Merge (APM), an activation-guided framework for cross-scale fusion. APM constructs task-conditioned activation maps on the donor, selects salient layers, hidden dimensions, attention heads, and MLP neurons to prune it to the recipient architecture, and injects the resulting donor slice into the original recipient using a micro interpolation coefficient. This formulation treats the donor as a source of concentrated functional components rather than requiring precise structural transplantation. Across 16 benchmarks spanning reasoning, mathematics, code generation, instruction following, and classification, APM improves the overall average accuracy from 55.5% to 60.6% over the original 3B recipient. RTE accuracy increases from 64.3% to 82.3%, QNLI from 52.3% to 65.7%, and BoolQ from 70.8% to 79.2%. Analyses of injection ratios and sequential multi-stage fusion further suggest that activation-guided extraction improves the quality of the transferable donor slice while preserving the small-ratio fusion regime. These results provide evidence that cross-scale heterogeneous fusion can succeed without explicit semantic alignment when the donor contribution is sufficiently concentrated and carefully selected.

Comments: 9 pages, 3 figures, and 7 tables

Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

Cite as: arXiv:2608.13596 [cs.LG]

(or arXiv:2608.13596v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2608.13596

arXiv-issued DOI via DataCite

Submission history

From: Jiahe Fan [view email] [v1] Fri, 31 Jul 2026 03:51:09 UTC (1,309 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning, by Jiahe Fan and 4 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.LG

new | recent | 2026-08

Change to browse by:

cs cs.AI

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

IArxiv recommender toggle

IArxiv Recommender (What is IArxiv?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • APM transfers capability from a large donor to an architecturally mismatched small recipient without training
  • Donor pruning is guided by task-conditioned activation maps, then merged at a very small interpolation ratio
  • Average accuracy over 16 benchmarks rises from 55.5% to 60.6%, with RTE improving from 64.3% to 82.3%
  • Results suggest cross-scale fusion can work without explicit neuron-wise semantic alignment

Highlights and analysis are generated automatically and may contain errors. Check the original source.