2026-07-24 13:02 UTCIn-site rewrite5 min readUpdated: 2026-07-24 13:44 UTC

Language Model Hallucination Evaluation with GraphEval

GraphEval is a hallucination evaluation framework that uses knowledge graphs and natural language inference to detect and localize factual inconsistencies in LLM outputs. This article provides a simulated, lightweight code example to illustrate its two-stage process: constructing a knowledge graph from the model's response, then evaluating each triple against a ground-truth context with an NLI model.

SourceKDnuggetsAuthor: Iván Palomares Carrascosa

--> Language Model Hallucination Evaluation with GraphEval - KDnuggets

-->

Join Newsletter

Introduction

Hallucinations are one of the best-known problems that large language models (LLMs) may experience when generating responses. They occur when a model produces a response that is factually incorrect, nonsensical, or simply made up, typically due to the model's lack of internal knowledge on the matter.

While many solutions have arisen in recent years to tackle the problem of model hallucinations, methodological evaluation frameworks for internally diagnosing them have been comparatively less studied. One recent study by Amazon researchers proposes using knowledge graphs as a means to analyze and detect hallucinations occurring in LLMs. The framework presented in the study is named GraphEval.

In this article, we will take a gentle, practical approach to illustrate the conceptual building blocks of GraphEval through a simulation-based, lightweight code example that you can easily try on your machine.

GraphEval in a Nutshell

GraphEval leverages knowledge graphs to identify and signal hallucinations in LLM-generated outputs. Unlike classical performance metrics that provide single scores to evaluate aspects like accuracy, certainty, and so on, GraphEval applies a two-stage evaluation process that emphasizes explainability, namely, providing insights into where exactly the hallucination took place.

To do this, GraphEval considers two stages:

Constructing a knowledge graph from the generated model response. The graph consists of semantic triples of the form (Subject, Relationship, Object), where subjects and objects correspond to nodes, and relationships correspond to the edges connecting those nodes.

Evaluating each triple in the constructed knowledge graph against a source context (a ground-truth body of knowledge) through a natural language inference (NLI) model. Any triple that cannot be entailed by the context according to the NLI engine — because it is contradictory or neutral — is flagged as a hallucination.

Illustrating GraphEval Through a Code Example

Before starting the code that simulates the application of the GraphEval framework, let's make sure we have the necessary libraries installed:

!pip install -q transformers networkx matplotlib torch

The purpose of the code example we are about to walk through is to demystify how the GraphEval methodology works, so we will replace the stages that would demand a heavy computational burden in a real-world setting with simulated, lightweight alternatives.

Accordingly, we will simulate a ground-truth knowledge base (context) assumed to contain factual information. In a production setting, this ground-truth knowledge would stem, for instance, from retrieving relevant documents from the vector database of a retrieval-augmented generation (RAG) system. For simplicity, here we directly create a ground-truth context and store it in source_context.

The ground-truth context provided to the LLM

source_context = ( "GraphEval is a hallucination evaluation framework based on representing information " "in Knowledge Graph (KG) structures. It acts as a pre-processing step and utilizes " "out-of-the-box NLI models to detect factual inconsistencies." )

Now, let's suppose the following is the original LLM response to a user prompt like "explain succinctly what GraphEval is". To initiate the first stage of the evaluation process, we would ask an auxiliary LLM to build the knowledge graph from that response. Both the response and the follow-up prompt used to obtain the knowledge graph are shown below:

The generated response we want to evaluate (contains a hallucination)

llm_output = ( "GraphEval is an evaluation framework that uses Knowledge Graphs. " "It requires a highly expensive, enterprise-level server farm to operate." )

Prompt template that would theoretically be passed to a local/free LLM (e.g. Mistral-7B)

KG_EXTRACTION_PROMPT = f""" You are an expert information extractor. Extract the core information from the following text as a Knowledge Graph. Return the output strictly as a Python list of tuples in the format: (Subject, Relationship, Object).

Text: {llm_output} """

Once again, for the sake of simplicity and to bypass the otherwise heavy computational load of running a massive LLM locally, let's suppose the following graph triples are obtained:

Simulated extraction to bypass the heavy computational load of running a massive LLM locally

extracted_triples = [ ("GraphEval", "is", "evaluation framework"), ("GraphEval", "uses", "Knowledge Graphs"), ("GraphEval", "requires", "expensive enterprise server farm") ]

print("Extracted Triples:") for t in extracted_triples: print(t)

Output:

Extracted Triples: ('GraphEval', 'is', 'evaluation framework') ('GraphEval', 'uses', 'Knowledge Graphs') ('GraphEval', 'requires', 'expensive enterprise server farm')

We deliberately added a triple that is fundamentally a hallucination (no enterprise server farm needed whatsoever!), so we can demonstrate how the subsequent NLI process applied to the knowledge graph reveals it.

Enough simulated steps for today. Let's get into the real action for the next stage: the NLI process. The next piece of code is fundamental to leveraging the ideas behind GraphEval. It uses a pre-trained NLI model from Hugging Face — the model is publicly available, so no access token is needed to download it — to compare each triple against the ground-truth context. If no entailment is "predicted" by the NLI model for a given triple, it is labeled as a hallucination.

from transformers import pipeline

Loading the open-source NLI model

print("Loading DeBERTa NLI model...") nli_evaluator = pipeline("text-classification", model="cross-encoder/nli-deberta-v3-small")

def evaluate_triple(context, triple): subject, relation, obj = triple hypothesis = f"{subject} {relation} {obj}"

Checking if the context entails the hypothesis

result = nli_evaluator({"text": context, "text_pair": hypothesis})

NLI models normally output: 'entailment', 'neutral', or 'contradiction'

label = result['label'].lower()

In GraphEval, anything other than 'entailment' is flagged as a hallucination

is_hallucinated = label != 'entailment'

return is_hallucinated, label, hypothesis

Running the evaluation pipeline

evaluation_results = []

print("\n--- GraphEval Results ---") for t in extracted_triples: is_hallucinated, nli_label, hypothesis = evaluate_triple(source_context, t) evaluation_results.append((is_hallucinated, nli_label))

status = "🚨 HALLUCINATION" if is_hallucinated else "✅ GROUNDED" print(f"{status} | Triple: {t} | NLI Output: {nli_label}")

Output:

--- GraphEval Results --- ✅ GROUNDED | Triple: ('GraphEval', 'is', 'evaluation framework') | NLI Output: entailment ✅ GROUNDED | Triple: ('GraphEval', 'uses', 'Knowledge Graphs') | NLI Output: entailment 🚨 HALLUCINATION | Triple: ('GraphEval', 'requires', 'expensive enterprise server farm') | NLI Output: neutral

As we expected, the last triple in the knowledge graph is detected as a hallucination.

To finish with a visual touch, we can also display the knowledge graph of the original LLM response alongside the detection results:

import networkx as nx import matplotlib.pyplot as plt import matplotlib.patches as mpatches

def visualize_grapheval(triples, eval_results): G = nx.DiGraph() edge_colors = []

for (triple, res) in zip(triples, eval_results): sub, rel, obj = triple is_hallucinated = res[0]

G.add_node(sub) G.add_node(obj) G.add_edge(sub, obj, label=rel)

Color-code the edges based on the NLI evaluation

edge_colors.append('red' if is_hallucinated else 'green')

Set up the plot

plt.figure(figsize=(10, 6)) pos = nx.spring_layout(G, seed=42)

Draw nodes

nx.draw_networkx_nodes(G, pos, node_color='lightblue', node_size=2500) nx.draw_networkx_labels(G, pos, font_size=10, font_weight='bold')

Draw edges and labels

nx.draw_networkx_edges(G, pos, edge_color=edge_colors, width=2.5, arrowsize=20) edge_labels = nx.get_edge_attributes(G, 'label') nx.draw_networkx_edge_labels(G, pos, edge_labels=edge_labels, font_color='black')

Add legend

green_patch = mpatches.Patch(color='green', label='Grounded (Entailment)') red_patch = mpatches.Patch(color='red', label='Hallucination (Neutral/Contradiction)') plt.legend(handles=[green_patch, red_patch], loc='lower right')

plt.title("GraphEval Hallucination Map", fontsize=14, fontweight='bold') plt.axis('off') plt.tight_layout() plt.show()

Render the knowledge graph

visualize_grapheval(extracted_triples, evaluation_results)

Resulting visualization:

Closing Remarks

GraphEval is an evaluation methodology proposed to help detect and localize the root cause of hallucinations in LLM outputs. This article turned its key principles and methodological stages into a simulated practical scenario to better understand its usefulness and its key implications for potential implementation in production systems.

Iván Palomares Carrascosa is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world.

Our Top 5 Free Course Recommendations

-->