Review Arcade: On the Human Alignment and Gameability of LLM Reviews
This paper empirically evaluates LLM-generated reviews for scientific papers, finding limited alignment with human reviews that varies significantly across prompts and models. It also shows that authors can game the system by iteratively revising papers based on LLM feedback, achieving statistically significant score increases for up to 35% of papers.
Article intelligence
Key points
- LLM reviews show limited alignment with human reviews
- Alignment quality varies substantially across different prompts and models
- Authors can game LLM reviews through iterative draft-revise workflows
- Up to 35% of papers see statistically significant score improvements via gaming
Why it matters
This matters because LLM reviews show limited alignment with human reviews.
Technical impact
May affect model selection, inference cost, product capability, and evaluation benchmarks.
[2605.28897] Review Arcade: On the Human Alignment and Gameability of LLM Reviews
[Submitted on 27 May 2026]
Title:Review Arcade: On the Human Alignment and Gameability of LLM Reviews
View a PDF of the paper titled Review Arcade: On the Human Alignment and Gameability of LLM Reviews, by Hans Ole Hatzel and 2 other authors
View PDF HTML (experimental)
Abstract:LLM-generated reviews for scientific papers are gaining considerable traction and are even being officially piloted by major conferences. We have to assume that not only reviewers are using LLM-assistance, but also that authors use LLMs to revise their papers before submitting. In this work, we perform empirical experiments on papers from the 2025 ACL Rolling Review (ARR) to evaluate LLM reviews from both the author and the reviewer perspective. First, we identify a limited alignment of LLM reviews with human ones. In the best-case scenario, the alignment is reasonable. However, we also find that LLM-human alignment varies substantially across prompts and models. Finally, we investigate the scenario in which the author uses an iterative draft-revise workflow to improve the submission according to the LLM review. We find that this "gaming" of LLM reviews can be effective in specific scenarios, leading to a statistically significant increase of overall scores for up to 35\% of papers. We publish our code: this https URL.
Comments: Under Review EMNLP 26
Subjects:
Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
Cite as: arXiv:2605.28897 [cs.AI]
(or arXiv:2605.28897v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2605.28897
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Jan Strich [view email] [v1] Wed, 27 May 2026 12:40:35 UTC (187 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Review Arcade: On the Human Alignment and Gameability of LLM Reviews, by Hans Ole Hatzel and 2 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new | recent | 2026-05
Change to browse by:
cs cs.MA
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)