Skip to content
AI News HubLIVE
More
Summary only

We taught a small LLM to throw away 68% of our RAG context

Kapa.ai introduced a small LLM pruner between retriever and generator in their RAG pipeline that grades retrieved chunks on a 5-level scale, discarding about 68% of context while preserving 96% recall, resulting in a net 34% cost reduction per query.

SourceHacker News AIAuthor: emil_sorensen
We taught a small LLM to throw away 68% of our RAG context
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions

Summary only

Under the source content permission, this page provides an editorial summary, key points, and a link to the original article.

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • Added a pruning step with a small LLM between retriever and generator to grade and discard unnecessary chunks.
  • Uses a 5-level scoring system that evaluates all chunks together, solving the set relevance issue of pointwise scoring.
  • Discards ~68% of context, retains 96% recall, and cuts cost by ~34% net.
  • Pruner adds ~0.7s latency, marginal inside agents; enabled by default in Product Agent SDK.

Highlights and analysis are generated automatically and may contain errors. Check the original source.

More recent stories on this topic · Models