2026-05-20原文2 min readUpdated: 2026-06-12

UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing

UCCI is a calibration-first router that maps token-level margin uncertainty to per-query error probability via isotonic regression and selects escalation threshold by constrained cost minimization. On a production NER workload, UCCI reduces inference cost by 31% at micro-F1=0.91 while reducing ECE from 0.12 to 0.03.

SourcearXiv Machine LearningAuthor: Varun Kotte

[2605.18796] UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing

[Submitted on 11 May 2026]

Title:UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing

View a PDF of the paper titled UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing, by Varun Kotte

View PDF HTML (experimental)

Abstract:LLM cascades and model routing promise lower inference cost by sending easy queries to a small model and escalating hard ones to a large model, but most deployed routers use uncalibrated confidence scores and require per-workload threshold tuning. We present UCCI, a calibration-first router that maps token-level margin uncertainty to a per-query error probability via isotonic regression and selects the escalation threshold by constrained cost minimization. Under three explicit assumptions, threshold policies on the calibrated score are cost-optimal, and isotonic calibration achieves O(n^{-1/3}) sample complexity for expected calibration error (ECE). On a production named entity recognition workload of 75,000 queries served by 4B and 12B instruction-tuned LLMs on H100 GPUs, UCCI cuts inference cost by 31% (95% CI: [27%, 35%]) at micro-F1 = 0.91 while reducing ECE from 0.12 to 0.03. At the same operating point, UCCI beats entropy thresholding, split-conformal routing, and a FrugalGPT-style learned threshold. All cascade results use end-to-end routing on actual model outputs and measured H100 latency, not simulated routing from global accuracies or nominal API prices.

Comments: 9 pages, 2 figures, 4 tables. Code: this https URL

Subjects:

Machine Learning (cs.LG); Computation and Language (cs.CL)

Cite as: arXiv:2605.18796 [cs.LG]

(or arXiv:2605.18796v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2605.18796

arXiv-issued DOI via DataCite

Submission history

From: Varun Kotte [view email] [v1] Mon, 11 May 2026 07:06:57 UTC (197 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing, by Varun Kotte

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.LG

new | recent | 2026-05

Change to browse by:

cs cs.CL

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)