The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Models
This paper introduces 'constraint tax,' a metric for the accuracy loss caused by structured output constraints in small language models. Experiments show that enforcing schemas like JSON increases validity but reduces answer accuracy, advocating for a 'reason free, constrain late' approach. Production systems should report multiple metrics separately.
Article intelligence
Key points
- Hard output constraints impose a 'constraint tax,' lowering answer accuracy for small models.
- Experiments show schema validity rose from 61.5% to 100%, but answer accuracy fell from 19.7% to 11.0%.
- Wrong-valid-schema outputs increased from 49.5% to 88.9%.
- Recommends a 'reason free, constrain late' design pattern and separate reporting of metrics.
Why it matters
This matters because hard output constraints impose a 'constraint tax,' lowering answer accuracy for small models.
Technical impact
May affect model selection, inference cost, product capability, and evaluation benchmarks.
[2605.26128] The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Models
[Submitted on 20 May 2026]
Title:The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Models
View a PDF of the paper titled The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Models, by Jaideep Ray
View PDF HTML (experimental)
Abstract:Production LLM systems increasingly require machine-readable outputs: JSON objects, typed traces, regex-constrained fields, and tool-call schemas. This paper targets on-device and low-cost small language model (SLM) deployments, where sub-3B models are attractive for privacy, latency, and commodity hardware but have limited capacity to satisfy schemas while solving tasks. The usual engineering assumption is that hard output constraints improve reliability without changing the underlying answer. We show that this assumption is unsafe for small models. We introduce \emph{constraint tax}, a measurement protocol for isolating the answer and executable-accuracy loss caused by structured-output constraints at fixed model, fixed task distribution, and fixed problem instances. Across 15,000 commodity-GPU generations with Qwen2.5-0.5B, Qwen2.5-1.5B, and SmolLM2-1.7B, hard answer-only schema decoding raises schema validity from 61.5\% to 100.0\%, but lowers answer accuracy from 19.7\% to 11.0\% and increases wrong-valid-schema outputs from 49.5\% to 88.9\%. The strongest industry analogue is a deterministic calendar tool-call task: Qwen2.5-1.5B achieves 91.5\% executable accuracy with prompt-only JSON but only 48.0\% under the same hard tool-call schema, while both modes are 100.0\% schema-valid. The error is semantic, not structural. We also show that the 3B boundary still pays a direct-schema tax and that delayed packaging supports a constructive design pattern: reason free, constrain late. The practical conclusion is direct: production systems should report schema validity, answer accuracy, executable accuracy, and wrong-valid-schema rate separately.
Subjects:
Machine Learning (cs.LG); Software Engineering (cs.SE)
Cite as: arXiv:2605.26128 [cs.LG]
(or arXiv:2605.26128v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.26128
arXiv-issued DOI via DataCite
Submission history
From: Jaideep Ray [view email] [v1] Wed, 20 May 2026 07:11:32 UTC (1,423 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Models, by Jaideep Ray
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.LG
new | recent | 2026-05
Change to browse by:
cs cs.SE
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
IArxiv recommender toggle
IArxiv Recommender (What is IArxiv?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)