2026-06-18原文2 min readUpdated: 2026-06-18

Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text

Large language models (LLMs) are increasingly used for clinical text tasks, but their ability to preserve diagnostic uncertainty is underexplored. This paper introduces a benchmark of 1,200 clinical documents with 9,184 uncertainty annotations across five levels, and evaluates three LLMs. Results show that LLMs preserve original uncertainty cues less than half the time and struggle with nuanced distinctions between adjacent levels, highlighting a failure mode not captured by standard metrics.

SourcearXiv Computational LinguisticsAuthor: Hongbo Du, Zixin Lu, Jiaming Qu

[2606.18471] Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text

[Submitted on 16 Jun 2026]

Title:Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text

View a PDF of the paper titled Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text, by Hongbo Du and 2 other authors

View PDF HTML (experimental)

Abstract:Large language models (LLMs) are increasingly used for clinical text tasks such as summarization and revision. While most studies evaluate the fluency and coherence of LLM-generated text, whether LLMs correctly preserve diagnostic uncertainty remains underexplored. In clinical practice, phrases such as ``possible pneumonia'' communicate the strength of available evidence and directly guide decisions about follow-up testing and treatment. Altering these uncertainty expressions can change the clinical meaning entirely. In this paper, we systematically evaluated this problem in two steps. First, we constructed a benchmark of 1,200 clinical documents with 9,184 uncertainty annotations across five levels. Second, we evaluated three LLMs on this benchmark. Our results show that (1) LLMs preserve the original uncertainty cues poorly, often less than half the time; (2) LLMs struggle with nuanced distinctions between adjacent levels. This work reveals a failure mode not captured by standard evaluation metrics and provides implications for the safe deployment of LLMs in clinical workflows.

Subjects:

Computation and Language (cs.CL)

Cite as: arXiv:2606.18471 [cs.CL]

(or arXiv:2606.18471v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2606.18471

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Jiaming Qu [view email] [v1] Tue, 16 Jun 2026 20:30:53 UTC (729 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text, by Hongbo Du and 2 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-06

Change to browse by:

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)