2026-07-08 04:00 UTCOriginal source3 min readUpdated: 2026-07-08 08:19 UTC

The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment

A new study decomposes the yes-no bias in LLMs using crossed symmetrization, finding that frontier models' internal moral stance is nearly format-invariant, while Claude models show significant order and lexical biases, GPT-5.5 and Gemini near zero. Bias shrinks with extended reasoning and follows surface wording, not the verdict.

SourcearXiv Computational LinguisticsAuthor: Haonan Huang

Article intelligence

EngineersAdvanced

Key points

LLMs' yes-no bias in moral dilemmas can be decomposed into order bias (toward last option) and lexical pull (toward 'no'), but the underlying moral scale is format-invariant.
Claude models exhibit substantial bias (story-averaged -0.32 to -0.86), while GPT-5.5 and Gemini are near zero; extended reasoning reduces the artifact.
Swapping labels separates lexical and verdict attachments, revealing that the logical bias is near zero across frontier models; models follow the printed surface, not the verdict.

Why it matters

This matters because lLMs' yes-no bias in moral dilemmas can be decomposed into order bias (toward last option) and lexical pull (toward 'no'), but the underlying moral scale is format-invariant.

Technical impact

May affect model selection, inference cost, product capability, and evaluation benchmarks.

This panel is AI-generated and reviewed for accuracy.

-->

[Submitted on 6 Jul 2026]

Title:The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment

View a PDF of the paper titled The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment, by Haonan Huang

View PDF HTML (experimental)

Abstract:Large language models (LLMs) increasingly issue judgments read as binary verdicts, and a growing literature reports such judgments shifting under logically irrelevant changes of wording - among them an amplified yes-no bias on moral dilemmas, absent in humans. A single framing cannot say what such a shift is: in a yes/no question the word "no" is at once logical verdict, lexical token, and last-printed option. We introduce a psychometric battery that separates these: crossed symmetrization - every logically irrelevant factor flipped in balanced pairs - across a corpus of question forms. A graded rating across logically equivalent forms recovers a coherent internal moral scale: frontier models' stance $\theta$ is nearly format-invariant (cross-form incoherence 0.12-0.21 on a $\pm 1$ axis); small open-weight models fail in model-specific ways. Forcing the verdict through yes/no overlays a decomposable artifact: an order bias toward the last-printed option - opposite to classic human primacy - plus a lexical pull toward the word "no"; the artifact is substantial only in the Claude models (story-averaged -0.32 to -0.86), $\approx 0$ for GPT-5.5 and Gemini, and shrinks under extended reasoning. The word and the verdict share one token; swapping the words for arbitrary labels separates them, and the verdict-attached logical bias proves $\approx 0$ for every frontier model, while model-specific label and order attachments remain: the models are not drawn toward rejecting - the pull follows the printed surface, not the verdict it carries. A minimal model, $P = \sigma((\theta \pm m)/s)$, summarizes any such artifact by a framing susceptibility m and a moral decisiveness s, measurably distinct from sampling temperature. The battery applies unchanged to any dilemma set and binary format: measuring what a model values requires crossing the frames of the question, not asking once.

Subjects:

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)

Cite as: arXiv:2607.05552 [cs.CL]

(or arXiv:2607.05552v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2607.05552

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Haonan Huang [view email] [v1] Mon, 6 Jul 2026 18:37:43 UTC (2,243 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment, by Haonan Huang

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-07

Change to browse by:

cs cs.AI cs.CY

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)