2026-07-23 04:00 UTCOriginal source2 min readUpdated: 2026-07-23 08:35 UTC

Reference-Free Evaluation of Reasoning in Open-Ended Question Answering

A new framework decomposes LLM reasoning traces into segments, uses NLI and hypergraphs to audit reasoning, offering a more reliable reference-free evaluation than LLM-as-judge, validated on math and medical benchmarks.

SourcearXiv Computational LinguisticsAuthor: Guneet Singh Kohli, Yuxiang Zhou, Michael Sejr Schlichtkrull, Gregory E Dean, Maria Liakata

-->

[Submitted on 22 Jul 2026]

Title:Reference-Free Evaluation of Reasoning in Open-Ended Question Answering

View a PDF of the paper titled Reference-Free Evaluation of Reasoning in Open-Ended Question Answering, by Guneet Singh Kohli and 4 other authors

View PDF HTML (experimental)

Abstract:AI-generated answers in high-stakes domains are often fluent but difficult to verify, especially when they contain multi-step reasoning rather than a single final answer. We propose a reasoning-based, reference-free framework for auditing LLM-generated outputs. The method decomposes a generated reasoning trace into segments, labels local premise-target relations using Natural Language Inference (NLI), and organizes these relations into a hypergraph. A deterministic backward AND-OR search then assigns segment-level audit labels that indicate how each segment is grounded within the generated response. We evaluate the framework in two settings: deductive mathematical reasoning with Hard2Verify, and open-ended medical reasoning with UroReason, a new physician-annotated benchmark of LLM reasoning traces from real clinical cases. Across these settings, our NLI-hypergraph audit provides a more reliable reference-free evaluation signal than direct LLM-as-judge baselines. In the clinical setting, state-of-the-art LLM judges often fail to identify problematic reasoning segments, over-accepting fluent but weakly grounded responses. Our results show that QA evaluation should account for how inferential relations compose across a reasoning trace, rather than relying only on final answers or LLMs as verifiers. UroReason will be made available through an API, and our code will be released as open source.

Subjects:

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

Cite as: arXiv:2607.19678 [cs.CL]

(or arXiv:2607.19678v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2607.19678

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Guneet Singh Kohli [view email] [v1] Wed, 22 Jul 2026 02:27:51 UTC (1,299 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Reference-Free Evaluation of Reasoning in Open-Ended Question Answering, by Guneet Singh Kohli and 4 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-07

Change to browse by:

cs cs.AI cs.LG

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)