2026-07-01 04:00 UTCOriginal source2 min readUpdated: 2026-07-01 08:08 UTC

When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs

A new study introduces ACE, an evaluation framework that controls for accuracy differences when comparing LLM calibration, revealing that many previously observed calibration advantages vanish and rankings can reverse.

SourcearXiv Computational LinguisticsAuthor: Zhichao Yang, Caiqi Zhang, Ruihan Yang, Chengzu Li, Nigel Collier, Deqing Yang

Article intelligence

InvestorsAdvanced

Key points

Global calibration metrics like ECE and Brier Score are confounded by model accuracy, leading to unfair comparisons.
The proposed ACE framework provides three complementary calibration views: Instance-Aligned, Distribution-Aligned, and Candidate-Aligned.
Experiments show that many calibration advantages reported in prior work weaken substantially after accuracy control, and ranking reversals are common.
Fair calibration comparison requires accuracy-aware evaluation.

Why it matters

This matters because global calibration metrics like ECE and Brier Score are confounded by model accuracy, leading to unfair comparisons.

Technical impact

May affect model selection, inference cost, product capability, and evaluation benchmarks.

This panel is AI-generated and reviewed for accuracy.

-->

[Submitted on 29 Jun 2026]

Title:When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs

View a PDF of the paper titled When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs, by Zhichao Yang and 5 other authors

View PDF

Abstract:Calibration evaluates whether a model confidence aligns with its empirical accuracy. Existing studies often compare the calibration of different large language models using global calibration metrics such as Expected Calibration Error and Brier Score. We begin by showing, both theoretically and empirically, that such comparisons are confounded by differences in model accuracy. For fairer cross-model comparison, we then propose ACE, an accuracy-controlled evaluation framework with three complementary views: Instance-Aligned, Distribution-Aligned, and Candidate-Aligned calibration. Across multiple benchmarks, model families, and confidence elicitation methods, we use ACE to study two practically important comparison axes, small versus large models and thinking versus non-thinking models. We find that many previously reported calibration advantages under raw global metrics weaken substantially after accuracy control. We also find that ranking reversal is frequent: models favored by raw metrics often cease to be favored once accuracy is controlled. Our results show that raw global calibration metrics are not robust for cross-model comparison, and that fair calibration comparison requires accuracy-aware evaluation.

Subjects:

Computation and Language (cs.CL)

Cite as: arXiv:2606.30814 [cs.CL]

(or arXiv:2606.30814v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2606.30814

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Ruihan Yang [view email] [v1] Mon, 29 Jun 2026 18:37:48 UTC (9,358 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs, by Zhichao Yang and 5 other authors

View PDF

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-06

Change to browse by:

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)