Skip to content
AI News HubLIVE
Original source2 min read

Watch Where You Linearize: Predicting and Correcting Model-Merging Interference

Summary

The paper shows that interference in model merging can be predicted by a first-order directional derivative of task A's outputs along task B's vector, but only when evaluated at the receiver model. Based on this, LR-Merge, a label-free correction method, significantly improves various base mergers with high computational efficiency.

SourcearXiv Machine LearningAuthor: Ido Nitzan Hidekel, Dan Raviv
Watch Where You Linearize: Predicting and Correcting Model-Merging Interference
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[Submitted on 18 Nov 2026]

Title:Watch Where You Linearize: Predicting and Correcting Model-Merging Interference

View a PDF of the paper titled Watch Where You Linearize: Predicting and Correcting Model-Merging Interference, by Ido Nitzan Hidekel and Dan Raviv

View PDF HTML (experimental)

Abstract:Model merging combines separately fine-tuned models by adding their task vectors to a shared backbone. Interference is typically assessed only after constructing and evaluating a candidate merge, making coefficient search expensive. We show that a first-order directional derivative of task A's outputs along task B's vector can predict interference, but only when evaluated at the appropriate model.

At the receiver model, this derivative forecasts nearby merges per task and per example, halves the accuracy error of a no-change baseline, and identifies most examples whose predictions flip. For full multi-task merges, a single evaluation is insufficient; two evaluations along the merge path are needed. At the shared pretrained backbone, however, the derivative is nearly orthogonal to realized changes in our vision experiments and only partly predictive in LoRA language-model pilots. Controls link this failure to displacement along the receiver's own task vector, not simply parameter distance; the mismatch appears within the first few dozen fine-tuning steps.

Freezing receiver-local responses produces a surrogate affine in the merge coefficients. We use it to construct LR-Merge, a label-free correction that optimizes coefficients through convex teacher matching on unlabeled probes, without backpropagation through merged networks. LR-Merge improves all tested closed-form, isotropic, and sparsified base mergers, with gains generally increasing as more tasks are merged. Using pretrained rather than receiver-local responses instead degrades the base merge. LR-Merge matches exact teacher matching through a merged network, which takes roughly three times longer to converge and performs similarly at an equal compute budget. Once built, the surrogate answers new coefficient and task-subset queries in seconds.

Subjects:

Machine Learning (cs.LG)

Cite as: arXiv:2611.25000 [cs.LG]

(or arXiv:2611.25000v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2611.25000

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Ido Nitzan Hidekel [view email] [v1] Wed, 18 Nov 2026 16:57:20 UTC (133 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Watch Where You Linearize: Predicting and Correcting Model-Merging Interference, by Ido Nitzan Hidekel and Dan Raviv

View PDF

HTML (experimental)

TeX Source

view license

Additional Features

Audio Summary

Current browse context:

cs.LG

new | recent | 2026-11

Change to browse by:

cs

References & Citations

SciX

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

IArxiv recommender toggle

IArxiv Recommender (What is IArxiv?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

InvestorsAdvanced

Key points

  • First-order directional derivative at the receiver model predicts merging interference, halving accuracy error.
  • At the shared pretrained backbone, the derivative is nearly orthogonal to realized changes, linking failure to displacement along the receiver's task vector.
  • LR-Merge optimizes coefficients via convex teacher matching on unlabeled probes without backpropagation, improving closed-form, isotropic, and sparsified mergers.
  • The surrogate model answers new coefficient and task-subset queries in seconds.

Highlights and analysis are generated automatically and may contain errors. Check the original source.