Skip to content
AI News HubLIVE
Original source2 min read

MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring

Summary

A paper accepted to AACL-IJCNLP 2026 introduces MisconceptDialogue, a fully automated framework in which five open-weight 7B–14B LLM tutors work with simulated students across more than 5,000 dialogues to test whether a model can spot, diagnose, and remediate math errors. On identical dialogues, exact-match scoring reports a 42.1% remediation success rate while a human-calibrated LLM judge reports 81.3%, a 39.2-point gap larger than the combined effects of tutor model, teaching strategy, and diagnostic information. The tutors also barely change their scores when given a deliberately swapped misconception, suggesting they do not reliably condition on the diagnosis they receive.

SourcearXiv Computational LinguisticsAuthor: Woojin Lee, Yoonkyu Woo, Jinah Kim, Hyeok-Min Gwon, Jin-Xia Huang
MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[Submitted on 1 Oct 2026]

Title:MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring

View a PDF of the paper titled MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring, by Woojin Lee and 4 other authors

View PDF HTML (experimental)

Abstract:Large language models are increasingly used as math tutors, raising two questions: can a single LLM detect a student's error, diagnose the underlying misconception, and remediate it through multi-turn dialogue - and can we trust the numbers that report how well it does? We address both with MISCONCEPTDIALOGUE, a fully automated framework that runs the full pipeline on five open-weight LLM tutors (7B-14B) interacting with LLM-simulated students over 5,000+ dialogues - a deliberately simulated protocol (paired conditions, a wrong-diagnosis control unsafe with real learners, scale) whose results do not establish real-student learning. On measurement, our headline finding is uncomfortable for the field: benchmark-style exact-match scoring common in standard math NLP - a lower-bound stress test - reports 42.1% measured Remediation Success Rate on identical dialogues where a human-calibrated LLM-as-judge reports 81.3%, a 39.2 percentage-point spread larger than the combined effects of tutor model, teaching strategy, and supplied diagnostic information. This is not a single-judge artifact: four independent LLM judges and 200 human annotations converge to within a few points (kappa up to 0.91). On capability, the verdict is equally striking: the score barely changes whether we tell the tutor the student's actual misconception or a deliberately swapped one - under the tested diagnosis-presentation format, these tutors show little evidence of reliably conditioning on the diagnostic information they are handed. For LLM tutoring evaluation to mature, multi-criterion scoring with multi-judge convergence and human anchoring should become standard practice - and the measurement question deserves as much scrutiny as the capability question it claims to answer.

Comments: Accepted to the AACL-IJCNLP 2026 Main Conference. 20 pages, 1 figure, 16 tables

Subjects:

Computation and Language (cs.CL)

Cite as: arXiv:2610.31679 [cs.CL]

(or arXiv:2610.31679v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2610.31679

arXiv-issued DOI via DataCite

Submission history

From: Woojin Lee [view email] [v1] Thu, 1 Oct 2026 05:48:24 UTC (465 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring, by Woojin Lee and 4 other authors

View PDF

HTML (experimental)

TeX Source

view license

Additional Features

Audio Summary

Current browse context:

cs.CL

new | recent | 2026-10

Change to browse by:

cs

References & Citations

SciX

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

InvestorsAdvanced

Key points

  • MisconceptDialogue evaluates five 7B–14B open-weight tutors over 5,000+ simulated dialogues; the authors stress this protocol does not establish real-student learning.
  • Exact-match scoring yields 42.1% remediation success versus 81.3% from a human-calibrated LLM judge on the same dialogues, a 39.2-point spread.
  • Four independent LLM judges and 200 human annotations converge within a few points (kappa up to 0.91), so the gap is not a single-judge artifact.
  • Scores barely change when the tutor is given the student's actual misconception versus a deliberately swapped one.
  • The authors argue multi-criterion scoring with multi-judge convergence and human anchoring should become standard for LLM tutoring evaluation.

Highlights and analysis are generated automatically and may contain errors. Check the original source.

More recent stories on this topic · Models