Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges
LLM-as-judge evaluations assume stable judgments, but this paper shows they can be manipulated through post-decision interaction. Experiments on MT-Bench and AlpacaEval reveal that while judges are stable under neutral reevaluation, targeted challenges can reverse decisions, affecting rankings and human agreement. The paper introduces the Evaluation Robustness Score (ERS).
[2606.05384] Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges
[Submitted on 3 Jun 2026]
Title:Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges
View a PDF of the paper titled Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges, by Srimonti Dutta and Akshata Kishore Moharir
View PDF HTML (experimental)
Abstract:LLM-as-judge evaluation is widely used in benchmarking pipelines, where model outputs are compared and ranked using automated evaluators. These pipelines typically assume that judgments are stable properties of fixed inputs. We show that this assumption does not hold under interaction. We study post-decision manipulability: the extent to which an evaluation outcome can be altered through subsequent conversation with the judge after an initial decision has been made. Across controlled experiments on MT-Bench and AlpacaEval, we find that LLM judges are highly stable under repeated and neutral reevaluation, yet become substantially reversible under targeted post-decision challenge. An anti-baseline challenge protocol shows that stable judgments can be overturned through motivated interaction, while a counterbalanced target-validation protocol separates this reversibility from net target-directed steering. These reversals have practical consequences: they can degrade agreement with human preferences, shift benchmark rankings, and produce harmful evaluation changes despite high self-reported confidence. Authority framing is especially destabilizing, and revised judgments are often accompanied by low-overlap justifications, suggesting post hoc rationalization rather than reliable error correction. We introduce the Evaluation Robustness Score (ERS) to quantify interactional robustness by combining reversal susceptibility with counterbalanced directional effects. Our findings identify post-decision interaction as a distinct failure mode for LLM-as-judge evaluation and motivate evaluation protocols that measure not only static agreement, but robustness under challenge.
Comments: Accepted at ACL 2026 GEM (Generation, Evaluation and Metrics) Workshop
Subjects:
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as: arXiv:2606.05384 [cs.AI]
(or arXiv:2606.05384v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.05384
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Srimonti Dutta [view email] [v1] Wed, 3 Jun 2026 19:37:23 UTC (121 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges, by Srimonti Dutta and Akshata Kishore Moharir
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new | recent | 2026-06
Change to browse by:
cs cs.CL
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)