Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking
This work fine-tunes LLaMA 3 (8B) as an efficient drop-in reranker via supervised fine-tuning and 4-bit quantization, replacing cross-encoders in RAG pipelines. It achieves 14-21% improvement in answer relevancy, context precision, answer similarity, and answer correctness on a domain-specific QA benchmark while reducing inference overhead.
-->
[Submitted on 11 Jul 2026]
Title:Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking
View a PDF of the paper titled Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking, by Shreeya Dasa Lakshminath and 1 other authors
View PDF HTML (experimental)
Abstract:Cross-encoders achieve high reranking accuracy in Retrieval-Augmented Generation (RAG) pipelines but impose quadratic inference costs that limit real-time deployment. We address this by fine-tuning LLaMA 3 (8B) as a drop-in reranker using a two-stage pipeline: supervised fine-tuning on a custom query-document relevance dataset via the Unsloth framework with LoRA adapters, followed by 4-bit quantization for efficient inference. The resulting model replaces the cross-encoder in a dual-retriever RAG pipeline combining BM25 and dense vector search. Evaluated on a domain-specific question-answering benchmark using the RAGAS framework, our fine-tuned LLaMA 3 reranker achieves gains of 14% in answer relevancy, 16% in context precision, 19% in answer similarity, and 21% in answer correctness over the cross-encoder baseline, while reducing inference overhead through 4-bit quantization. These results demonstrate that instruction-tuned LLMs can be adapted into accurate, efficient rerankers without the quadratic complexity of traditional cross-encoders.
Comments: 6 pages, 4 figures. This work was completed in 2024
Subjects:
Computation and Language (cs.CL); Information Retrieval (cs.IR); Machine Learning (cs.LG)
Cite as: arXiv:2607.11933 [cs.CL]
(or arXiv:2607.11933v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2607.11933
arXiv-issued DOI via DataCite
Submission history
From: Shreeya Dasa Lakshminath [view email] [v1] Sat, 11 Jul 2026 04:31:20 UTC (784 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking, by Shreeya Dasa Lakshminath and 1 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new | recent | 2026-07
Change to browse by:
cs cs.IR cs.LG
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)