RAG Evaluation Frameworks Compared: RAGAS vs TruLens vs DeepEval

-->

RAG Evaluation Frameworks: RAGAS vs TruLens vs DeepEval

India's Most Futuristic AI Conference Is Back – Bigger, Sharper, Bolder

d

:

h

:

m

:

s

Career

GenAI

Prompt Engg

ChatGPT

LLM

Langchain

RAG

AI Agents

Machine Learning

Deep Learning

GenAI Tools

LLMOps

Python

NLP

SQL

AIML Projects

Reading list

How to Become a Data Analyst in 2025: A Complete RoadMap

A Comprehensive Learning Path to Tableau in 2025

A Comprehensive NLP Learning Path 2025

Learning Path to Become a Data Scientist in 2025

Step-by-Step Roadmap to Become a Data Engineer in 2025

A Comprehensive MLOps Learning Path: 2025 Edition

Roadmap to Become an AI Engineer in 2025

A Comprehensive Learning Path to Master Computer Vision in 2025

Best Roadmap to Learn Generative AI in 2025

GenAI Roadmap for Enterprises

Large Language Models Demystified: A Beginner’s Roadmap

Learning Path to Become a Prompt Engineering Specialist

RAG Evaluation Frameworks Compared: RAGAS vs TruLens vs DeepEval

Soumil Jain Last Updated : 12 Jul, 2026

10 min read

LLMs are getting stronger every day, and building a RAG pipeline has never been easier. Knowing whether it actually works is not. Most teams ship a RAG system, see decent-looking answers, and call it done, until users hit hallucination, missing context, or irrelevant chunks.

That’s where evaluation frameworks come in. RAGAS, TruLens, and DeepEval are three of the most widely used tools for measuring RAG quality. In this article, I’ll break down how each one works and when to reach for it.

Table of contents

Why RAG Needs Its Own Evaluation Approach

The Three Layers of RAG Evaluation

Retrieval Metrics You Need to Know: Precision@K, Recall@K, MRR, NDCG

RAG Assessment

RAGAS

TruLens

DeepEval

Choosing Based on Where You Are

Common Pitfalls Across All Three

Conclusion

Frequently Asked Questions

Why RAG Needs Its Own Evaluation Approach

A RAG system has two moving parts: the retriever, which fetches context, and the generator, which writes the answer using that context. If either half fails, the final answer fails, but they fail in different ways. A bad retriever pulls irrelevant or incomplete chunks. A bad generator ignores good context and hallucinates anyway, or writes something technically correct but unhelpful.

Traditional NLP metrics like BLEU or ROUGE don’t capture any of this. They compare word overlap between the generated answer and a reference answer, useful for translation, not for judging whether an LLM stayed grounded in facts or whether the retriever did its job. RAG evaluation needs metrics that check both halves separately and together, without always requiring a handwritten correct answer for every query. This is exactly the gap RAGAS, TruLens, and DeepEval were built to fill, and each one approaches it with a different philosophy.

The Three Layers of RAG Evaluation

Before diving into each tool, it helps to know that most RAG metrics fall into three buckets:

Retrieval quality: Did the system fetch the right chunks?

Generation quality: Did the model use those chunks correctly, without hallucination.

End-to-End quality: Does the final answer satisfy the user’s questions?

Every framework below scores some combination of these three. The difference is in how automated scoring is whether it needs ground truth answers, and if the tool is meant for one-time evaluation or continuous monitoring.

Retrieval Metrics You Need to Know: Precision@K, Recall@K, MRR, NDCG

It is worth understanding the classic information – retrieval metrics that all three build on. These come from traditional search and recommendation systems, and they measure retrieval quality directly using labeled relevance data, no LLM judge needed.

Precision@K

What it measures: Out of the top K chunks your retriever returned, how many are actually relevant?

Formula: Precision@K = (Relevant chunks in top K) / K

Example: You retrieve the top 5 chunks for a query. 3 of them are relevant to answering it.

Then, Precision@5 = 3 / 5 = 0.6

A high Precision@K means your retriever isn’t pulling in noise. A low score means the LLM must sift through junk context, which increases hallucination risk since irrelevant chunks can confuse the generator.

Recall@K

What it measures: Out of all the relevant chunks that exist in your knowledge base, how many did you actually retrieve in the top K?

Formula: Recall@K = (Relevant chunks retrieved in top K) / (Total relevant chunks in the corpus)

Example: There are 4 relevant chunks in your entire corpus for a query. Your top 5 retrieved chunks contain 3 of them.

Then, Recall@5 = 3 / 4 = 0.75

Precision and Recall usually trade off against each other. Increasing K i.e increasing more chunks tends to raise Recall but lower Precision, since you’re pulling in more noise along with the useful chunks. This is why RAGAS reports context precision and context recall as two separate scores instead of one.

Mean Reciprocal Rank (MRR)

What it measures: How high up was the first relevant chuck in your ranked results? This matters because LLMs tend to pay more attention to earlier context, so burying the one useful chunk at position 8 is worse than having it at position 1

Formula: MRR = (1 / N) × Σ (1 / rank of first relevant chunk for each query)

Example: For Query A, the first relevant chunk is at rank 1 is

reciprocal rank = 1 / 1 = 1

For Query B, the first relevant chunk is at rank 3 then, the reciprocal rank = 1 / 3 = 0.3333

MRR = (1.0 + 0.33) / 2 = 0.665

Higher MRR means your retriever consistently surfaces the best chunk near the top, not buried deep in the results.

Normalized Discounted Cumulative Gain (NDCG)

What it measures: A more nuanced version of the above, it accounts for graded relevance, i.e some chunks are more relevant than others, not just relevant/irrelevant, and penalize relevant chunks that appear lower in the ranking.

How it works:

DCG (Discounted Cumulative Gain): It sums up relevance scores of retrieved chunks, but discounts chunks that appear lower in the ranking (using a log based penalty).

NDCG: Normalizes that DCG value against the ideal possible ranking i.e, if the retriever had perfectly ordered chunks from most to least relevant, producing a score between 0 and 1.

Formula:

An NDCG of 1 means your retriever ranked chunks exactly as well as theoretically possible. This metric matters more when relevance isn’t binary. For example: A chunk might be highly relevant, somewhat relevant or irrelevant rather than just yes or no.

How do these connect to the frameworks?

Keep these four in mind as you read the next sections, each framework re-implements a version of them, just scored differently:

Precision@K: It reappears as RAGAS’s Context Precision and DeepEval’s Contextual Precision but scored by an LLM judge instead of exact-match relevance labels.

Recall@K: Reappears as RAGAS’s Context recall and DeepEval’s Contextual Recall, checking whether all necessary information was retrieved.

MRR and NDCG: these don’t have a direct named equivalent in any of the three frameworks, but TruLen’s per chunk Context Relevance scores gives you the raw data to compute them yourself.

The main difference is that classic Information Retrieval (IR) metrics require pre-established relevance labels (“which chunks are relevant“) before you can compute anything. RAGAS, TruLens, and DeepEval swap in an LLM for that relevance judgment, avoiding the requirement. This needs little setup but adds more noise than a proper IR benchmark. To check each method’s trustworthiness, label 200 to 300 query/chunk pairs and compute Precision@K, Recall@K, and NDCG as a baseline.

Read more: 12 Important Model Evaluation Metrics

RAG Assessment

Here we’ll put the 3 frameworks to test:

RAGAS

RAGAS is a python framework built specifically for RAG pipelines. Its biggest selling point is that most of its metrics don’t need human labelled ground truth, it uses an LLM as a judge to score outputs against the retrieval context itself, which makes it fast to set up on an existing pipeline.

Core metrics:

Faithfulness: Is the response consistent with the fetched context, or is there any fabricated information? RAGAS breaks the response into individual statements and checks to see if every individual statement can be confirmed against the source chunks.

Answer Relevance: Does the response directly answer the question, or does it provide indirect answers that would still make factual sense?

Context Precision: Are the fetched chunks consistent with the answer, or is there any unrelated content added to the useful content?

Context Recall: Did the retrieval contain complete and required information, or was there any missing critical pieces? This requires an answer for reference.

Answer Correctness: How closely does the generated output resemble an established answer? This will use a combination of fact similarities and semantic similarities and will require a source answer to use for comparison.

Semantic Similarity: A lower intensity version of the previous, it verifies that the generated output and source output can be compared using the premise of embedding, as opposed to verifying individual claims against each respective output.

How it works in practice:

You pass RAGAS a dataset of {questions, retrieved_contexts, generated_answer} and optionally a reference answer, and it runs each metric using an LLM under the hood.

A typical run looks like:

Code:

from ragas import evaluate from ragas.metrics import faithfulness, answer_relevancy, context_precision

results = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision])

Each metrics returns a score between 0 and 1, and RAGAS aggregates them into a summary table you can log or compare across pipeline version.

Synthetic test set generation: One of RAGAS’s most useful features that often gets overlooked, it can auto generate a test dataset directly from your documents instead of you writing questions and reference answers by hand. It samples chunks from your knowledge base, then uses an LLM to generate realistic questions, reference answers, and even harder variants. This solves the biggest bottleneck in RAG evaluation: building a labeled test set is normally the slowest part, and this cuts most of that manual work.

from ragas.testset import TestsetGenerator

generator = TestsetGenerator.from_langchain(llm, embeddings)

testset = generator.generate_with_langchain_docs( documents, testset_size=50 )

This gives you a ready-made {question, reference_answer, contexts} dataset to run every metric above against, without hand-labeling a single row.

Strengths: Minimal setup, works well for quick benchmarking, plays nicely with LangChain and LlamaIndex out of the box.

Limitations: Since most metrics rely on an LLM judge, scores can vary slightly run to run, and you’re paying for extra LLM calls on every evaluation pass.

Best for: Teams wanting fast, automated, reference-free scoring right after building a RAG prototype, or for A/B testing two versions of a retriever or prompt.

TruLens

TruLens takes a different angle, it’s built for observability, not just one-off scoring. Instead of running an evaluation once and getting a report, TruLens instruments your RAG app so it logs every step: what was retrieved, what the LLM saw, and what it generated. That log becomes the basis for its scores.

Core concept: the RAG Triad

Context Relevance:how relevant is the retrieved context to the query? Scored chunk by chunk, not just for the whole retrieved set.

Groundedness: is the answer supported by the context, or does it introduce claims that aren’t backed by any retrieved chunk?

Answer Relevance: does the answer address the actual question the user asked?

What sets TruLens apart is the tooling around these three scores. It ships with a dashboard which is built on Streamlit where you can:

Compare multiple experi

[truncated for AI cost control]