2026-05-25原文

Reading Calibrated Uncertainty from Language Model Trajectories

A new method uses geometric features from layer-wise MLP updates to train a sparse linear probe for better uncertainty quantification in language models, outperforming standard MSP by up to 21 AURC points on selective abstention.

Article intelligence

EngineersAdvanced

Key points

Maximum softmax probability (MSP) is cheap but often miscalibrated for language model uncertainty.
Proposed method extracts 11 scale-invariant geometric features from per-layer MLP trajectories.
Sparse linear probe significantly outperforms MSP on selective abstention, with gains scaling with baseline miscalibration.
Features have closed-form geometric meaning, enabling interpretable analysis of where errors occur across layers.

Why it matters

This matters because maximum softmax probability (MSP) is cheap but often miscalibrated for language model uncertainty.

Technical impact

May affect model selection, inference cost, product capability, and evaluation benchmarks.

[2605.22864] Reading Calibrated Uncertainty from Language Model Trajectories

[Submitted on 19 May 2026]

Title:Reading Calibrated Uncertainty from Language Model Trajectories

View a PDF of the paper titled Reading Calibrated Uncertainty from Language Model Trajectories, by Aliai Eusebi and 5 other authors

View PDF HTML (experimental)

Abstract:The maximum softmax probability (MSP) represents a default approach when evaluating uncertainty quantification for language model generation with structured output. Although cheap, it is often miscalibrated. Methods that probe the model's internal activations feed raw hidden states into opaque classifiers, reading activations as static snapshots and leaving implicit the layer-wise trajectory by which a representation is formed. Yet, similar endpoints can arise from very different paths, and how evidence accumulates, reinforces, or reverses across depth might reveal uncertainty that final probabilities obscure. We extract eleven scale-invariant geometric features, tracing the cumulative path of per-layer MLP updates, and feed them to a sparse linear probe. The probe outperforms MSP under selective abstention, with gains scaling with baseline miscalibration up to 21 AURC points. Because every feature has a closed-form geometric meaning, the probe's coefficients trace how and where along depth errors take shape -- which layers commit prematurely, which contradict the running state, where trajectories drift away from their endpoint.

Subjects:

Machine Learning (cs.LG)

Cite as: arXiv:2605.22864 [cs.LG]

(or arXiv:2605.22864v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2605.22864

arXiv-issued DOI via DataCite

Submission history

From: Aliai Eusebi [view email] [v1] Tue, 19 May 2026 19:24:29 UTC (2,016 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Reading Calibrated Uncertainty from Language Model Trajectories, by Aliai Eusebi and 5 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.LG

new | recent | 2026-05

Change to browse by:

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)