Skip to content
AI News HubLIVE
Original source3 min read

Consistency Is Not Correction: Separating Confidence, Abstention, and Recovery in LLM Agents

Summary

The paper argues that ranking, selective prediction, and correction utility are separate claims about a confidence signal, derives a pre-experiment bound on when gated correction can be detected, and shows empirically that many reported gains are capped by measurement resolution rather than signal quality.

SourcearXiv Computational LinguisticsAuthor: Khushal Sethi
Consistency Is Not Correction: Separating Confidence, Abstention, and Recovery in LLM Agents
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[Submitted on 15 Oct 2026]

Title:Consistency Is Not Correction: Separating Confidence, Abstention, and Recovery in LLM Agents

View a PDF of the paper titled Consistency Is Not Correction: Separating Confidence, Abstention, and Recovery in LLM Agents, by Khushal Sethi

View PDF HTML (experimental)

Abstract:A controller that resamples an answer, escalates to a larger model, or abstains is usually justified by a confidence signal. Three claims about such a signal do not follow from one another: it ranks answers (discrimination), it supports withholding some of them (selective prediction), and it changes an answer for the better (correction utility). We give an exact decomposition of the third into gated recoveries minus gated harms, which bounds whether a comparison can succeed before it is run: a gate that only replaces a proposal can act only where the base policy was already wrong, so the initial error count caps the discordant pairs any correct gate can produce. When that cap falls below the count a paired test needs, the experiment cannot return a positive result for a working mechanism, nor distinguish one from a gate that never fired. We measure all three targets on two model scales and find the bound binds. In a small-model suite a perturbation-agreement verifier reaches within-policy AUROC 0.990 (95% interval [0.95, 1.00]) while every episode outcome matches single-sample decoding at 4.32 times the call count. In a controlled 840-episode study no verifier replaced the proposal on any of the 60 GSM8K decisions, so that comparison tested nothing; the entire GSM8K gain across six controllers is one task, and it is the task whose identical temperature-zero repeats disagreed, making it the one answer the service does not reproduce. On the diagnostic set the gate could reach four errors while the test needed six, so even a perfect gate had no significant result available. We report two checks that would have caught this before any budget was spent, and show that tie handling in rankings, tied selection cutoffs, and unequal output limits each move a headline number in the same dataset. The binding constraint on this literature is measurement resolution, not the choice of signal.

Subjects:

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)

Cite as: arXiv:2611.21651 [cs.CL]

(or arXiv:2611.21651v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2611.21651

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Khushal Sethi [view email] [v1] Thu, 15 Oct 2026 05:55:30 UTC (61 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Consistency Is Not Correction: Separating Confidence, Abstention, and Recovery in LLM Agents, by Khushal Sethi

View PDF

HTML (experimental)

TeX Source

view license

Additional Features

Audio Summary

Current browse context:

cs.CL

new | recent | 2026-11

Change to browse by:

cs cs.AI cs.MA

References & Citations

SciX

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • Separates three distinct claims about confidence signals: discrimination, selective prediction, and correction utility.
  • Gives an exact decomposition and a bound: the base policy's initial error count caps the discordant pairs any correct gate can produce.
  • A perturbation-agreement verifier reaches AUROC 0.990 (95% interval [0.95, 1.00]) in a small-model suite while matching single-sample decoding exactly at 4.32x the call count.
  • In an 840-episode study no verifier replaced the proposal on any of the 60 GSM8K decisions, so the comparison tested nothing.
  • Tie handling in rankings, tied selection cutoffs, and unequal output limits each move a headline number on the same dataset.

Highlights and analysis are generated automatically and may contain errors. Check the original source.

Consistency Is Not Correction: Separating Confidence, Abstention, and Recovery in LLM Agents | AI News Hub