Skip to content
AI News HubLIVE
Original source2 min read

Benchmarking LLM Competence on Logical Inference over Probability Operators

Summary

A new study introduces a benchmark for logical inference over probability operators (e.g., 'probably', 'might', 'must'), featuring 14,320 procedurally generated English prompts across 15 inference templates. Evaluating 29 models, only 9 exceed random chance, with most showing answer biases independent of logical form.

SourcearXiv Computational LinguisticsAuthor: Nayera Hasan, Jack Greff, Alvin Grissom II
Benchmarking LLM Competence on Logical Inference over Probability Operators
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[Submitted on 29 Jul 2026]

Title:Benchmarking LLM Competence on Logical Inference over Probability Operators

View a PDF of the paper titled Benchmarking LLM Competence on Logical Inference over Probability Operators, by Nayera Hasan and 2 other authors

View PDF HTML (experimental)

Abstract:Both expressions of uncertainty and inferences are ubiquitous in natural language, and valid inferences over natural-language expressions of uncertainty are necessary for not only everyday conversations but also for high-stakes domains such as medicine and law. While large language models are increasingly evaluated on logical reasoning tasks, disentangling principled, symbolic reasoning from clever surface-level pattern matching is fraught with difficulty. We introduce a benchmark for reasoning over probability operators--inference over sentences with gradable epistemic modals (e.g., probably, might, must) containing 14,320 procedurally-generated English prompts across fifteen inference templates, systematically varying question form, negation strategy, and surface content. Evaluating 29 models, we find that most show answer biases independent of the logical form, a systematic preference for Yes or No. We summarize this with a competence floor: the worse of a model's accuracy on Yes-correct and No-correct items. Only 9 of 29 models exceed random chance. We also test variations in question form, verb phrases/activity, and both the gender and origin of names used in the prompts, finding biases across every axis.

Comments: Under review

Subjects:

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

Cite as: arXiv:2607.27405 [cs.CL]

(or arXiv:2607.27405v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2607.27405

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Alvin Grissom II [view email] [v1] Wed, 29 Jul 2026 19:22:09 UTC (231 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Benchmarking LLM Competence on Logical Inference over Probability Operators, by Nayera Hasan and 2 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-07

Change to browse by:

cs cs.AI

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • The benchmark contains 14,320 English prompts systematically varying question form, negation strategy, and surface content.
  • Only 9 of 29 evaluated models exceed random chance on the proposed 'competence floor' metric.
  • Models exhibit systematic biases across every tested axis, including question form, verb phrases, and name gender/origin.

Highlights and analysis are generated automatically and may contain errors. Check the original source.