When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering
This paper introduces OGCaReBench, a retrieval-focused benchmark for evaluating LLMs on clinical questions beyond standard guidelines. Experiments show that even GPT-5.2 achieves only 56% accuracy, but retrieval augmentation boosts performance to 82%, highlighting the need for evidence-grounded reasoning in medicine.
[2605.21807] When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering
[Submitted on 20 May 2026]
Title:When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering
View a PDF of the paper titled When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering, by Doeun Lee and 13 other authors
View PDF HTML (experimental)
Abstract:Across medical specialties, clinical practice is anchored in evidence-based guidelines that codify best studied diagnostic and treatment pathways. These pathways routinely fall short for the long tail of real-world care not covered by guidelines. Most medical large language models (LLMs), however, are trained to encode common, guideline-focused medical knowledge in their parameters. Current evaluations test models primarily on recalling and reasoning with this memorized content, often in multiple-choice settings. Given the fundamental importance of evidence-based reasoning in medicine, it is neither feasible nor reliable to depend on memorization in practice. To address this gap, we introduce OGCaReBench, a free-form retrieval-focused benchmark aimed at evaluating LLMs at answering clinical questions that require going beyond typical guidelines. Extracted from published medical case reports and validated by medical experts, OGCaReBench contains long-form clinical questions requiring free-text answers, providing a systematic framework for assessing open-ended medical reasoning in rare, case-based scenarios. Our experiments reveal that even the best-performing baseline (GPT-5.2) correctly answers only 56% of our benchmark with specialized models only reaching 42%. Augmenting models with retrieved medical articles improves this performance to up to 82% (using GPT-5.2) highlighting the importance of evidence-grounding for real-world medical reasoning tasks. This work thus establishes a foundation for benchmarking and advancing both general-purpose and medical LLMs to produce reliable answers in challenging clinical contexts.
Comments: 34 pages, 20 figures
Subjects:
Computation and Language (cs.CL)
Cite as: arXiv:2605.21807 [cs.CL]
(or arXiv:2605.21807v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.21807
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Doeun Lee [view email] [v1] Wed, 20 May 2026 23:04:48 UTC (1,142 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering, by Doeun Lee and 13 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new | recent | 2026-05
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)