RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator
RankJudge is a novel benchmark generator for evaluating LLMs as judges in multi-turn conversations. It creates paired conversations where one has a single flaw injected into one turn, enabling unambiguous labeling and precise isolation of failure categories. Implemented across machine learning, biomedicine, and finance, it evaluated 21 frontier LLM judges and ranked them via the Bradley-Terry model. The approach also uses difficulty ratings to dynamically curate evaluations, reducing label noise.
[2605.21748] RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator
[Submitted on 20 May 2026]
Title:RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator
View a PDF of the paper titled RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator, by Zhenwei Tang and 5 other authors
View PDF HTML (experimental)
Abstract:As interactive LLM-based applications are created and refined, model developers need to evaluate the quality of generated text along many possible axes. For simpler systems, human evaluation may be practical, but in complicated systems like conversational chatbots, the amount of generated text can overwhelm human annotation resources. Model developers have begun to rely heavily on auto-evaluation, where LLMs are also used to judge generation quality. However, existing LLM-as-a-judge benchmarks largely focus on simple Q\&A tasks that do not match the complexity of multi-turn conversations. We introduce RankJudge, a benchmark generator for evaluating LLM-as-a-judge on multi-turn conversations grounded in reference documents. RankJudge creates pairs of conversations where one conversation has a single flaw injected into one turn. This construction allows paired conversations to be labeled unambiguously as better or worse, and precisely isolates failure categories to individual turns, enabling a strict joint correctness criterion for judging. We implement RankJudge across the domains of machine learning, biomedicine, and finance, evaluate 21 frontier LLM judges, and rank those judges via the Bradley-Terry model. Our formulation also allows ranking each conversation pair with difficulty ratings, which we use to dynamically curate the evaluation slice to reduce label noise, as confirmed via human annotation. We find that judge rankings are stable under partial observability, coarser correctness criteria, and an alternative random-walk rating algorithm.
Subjects:
Computation and Language (cs.CL)
Cite as: arXiv:2605.21748 [cs.CL]
(or arXiv:2605.21748v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.21748
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Zhenwei Tang [view email] [v1] Wed, 20 May 2026 21:20:01 UTC (814 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator, by Zhenwei Tang and 5 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new | recent | 2026-05
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)