Beyond LoRA: Is Sparsity-Induced Adaptation Better?
A new paper proposes Cheap LoRA (cLA) and a chained circulant variant c³LA, inducing sparsity in existing LoRA variants for parameter-efficient fine-tuning. Theoretical generalization error bounds are derived, and experiments across 11 fine-tuning methods, 10 pretrained models, and 14 datasets show that sparse structured column-space restriction remains competitive while reducing training time by up to 10% and peak GPU memory by up to 15%.
[2606.13767] Beyond LoRA: Is Sparsity-Induced Adaptation Better?
[Submitted on 11 Jun 2026]
Title:Beyond LoRA: Is Sparsity-Induced Adaptation Better?
View a PDF of the paper titled Beyond LoRA: Is Sparsity-Induced Adaptation Better?, by Elijah Cadenhead and Cristian McGee and Xin Li and El Houcine Bergou and Aritra Dutta
View PDF HTML (experimental)
Abstract:Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models. However, questions remain about the comparative generalizability of these approaches and how the structural restrictions on low-rank updates preserve effective adaptation performance. We present a historical framing, covering the past (full fine-tuning and original LoRA), the present (different variants of LoRA), and propose simpler, cheaper, parameter-efficient extensions by inducing sparsity within existing LoRA variants: Cheap LoRA (cLA), training a single low-rank factor with the other fixed (deterministically or, in its randomized variant, stochastically), and the chained circulant variant, ${c}^3$LA. We frame cLA as a structured instance of asymmetric LoRA, serving as a controlled column-subspace restriction of full fine-tuning. We derive information-theoretic generalization error bounds for these variants, marking one of the first endeavors in this area. Empirically, we evaluate 11 fine-tuning methods across 10 pre-trained models and 14 datasets, analyzing the fine-tuned models' performance and generalization using tools such as loss landscapes and spectral analysis. Despite the sensitivity of fine-tuned models to the pre-trained model, datasets, and other factors, our study suggests that restricting LoRA-based PEFT methods' adaptation to a sparse, structured column space remains competitive across tasks with their parameter-matched baselines while reducing up to 10% training time and peak GPU memory up to 15%, even with a naïve, non-optimized, sparse implementation. Our theoretical and empirical generalization measures provide a more consistent and principled approach to their cost-effective adaptation than commonly used analytical tools. Overview and code are available at: this https URL.
Comments: Overview of the paper and code can be found here: this https URL
Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Theory (cs.IT)
MSC classes: 68T07, 68T05, 68Q32, 90C30, 94A17
ACM classes: I.2.6; I.5.1; G.1.6; G.3
Cite as: arXiv:2606.13767 [cs.LG]
(or arXiv:2606.13767v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.13767
arXiv-issued DOI via DataCite
Submission history
From: Elijah Cadenhead [view email] [v1] Thu, 11 Jun 2026 17:59:01 UTC (11,265 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Beyond LoRA: Is Sparsity-Induced Adaptation Better?, by Elijah Cadenhead and Cristian McGee and Xin Li and El Houcine Bergou and Aritra Dutta
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.LG
new | recent | 2026-06
Change to browse by:
cs cs.AI cs.IT math math.IT
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
IArxiv recommender toggle
IArxiv Recommender (What is IArxiv?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)