DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models
DeFAb is a dataset and pipeline converting four decades of public knowledge bases into defeasible abduction benchmarks. A symbolic solver achieves 100% accuracy in under 50 microseconds, while the best frontier LLM reaches 65% and drops to 23.5% under robust evaluation. The benchmark uses polynomial-time verifiable checks to measure logical rigor and creativity. It releases 372,648+ instances, along with harder variants and a Lean 4/Mathlib creative variant.
[2606.18557] DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models
[Submitted on 17 Jun 2026]
Title:DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models
View a PDF of the paper titled DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models, by Patrick Cooper and 1 other authors
View PDF HTML (experimental)
Abstract:A rule-based logic solver resolves every instance in our benchmark in under 50 microseconds with 100% accuracy; the best frontier language model reaches 65% at best and drops to 23.5% under rendering-robust evaluation (worst case over four surface renderings). We introduce DeFAb (Defeasible Abduction Benchmark), a dataset and generation pipeline that converts four decades of publicly funded knowledge bases into formally grounded instances for defeasible abduction: constructing hypotheses that explain anomalies by overriding defaults while preserving unrelated expectations. Because every hypothesis must pass polynomial-time checks for valid derivation, conservativity, and minimality, DeFAb makes logical rigor the instrument for measuring creativity and theoretical reasoning, scoring the disciplined construction of theory revisions rather than fluent but theory-destroying prose. The pipeline pairs taxonomic hierarchies (OpenCyc, YAGO, Wikidata) with behavioral property graphs (ConceptNet, UMLS) to produce 372,648+ instances across 33.75M materialized rules from 18 sources, in three levels with polynomial-time verifiable gold standards. Four frontier models do not reliably internalize defeasible reasoning: rendering-robust Level 2 accuracy is 7.8-23.5%; chain-of-thought variance (~36 pp) exceeds any inter-model gap; and a matched contamination control isolates a +19.4 pp Level 3 gap. We further release DeFAb-Hard (a 235-instance Level 3 difficulty variant; best model 53.3% vs 100% symbolic) and CONJURE (a kernel-verified transformative-creativity variant of 560 Lean 4/Mathlib instances whose gold answers are definitions the proof kernel did not previously contain, judge-free verifier; a pilot finds zero novel concepts). The same verifier doubles as an exact reward for preference optimization (DPO, RLVR/GRPO). Released under MIT at this https URL.
Comments: 33 pages, 14 figures, 23 tables. Dataset: this https URL ; code and evaluation harness: this https URL
Subjects:
Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Logic in Computer Science (cs.LO)
MSC classes: 68T27, 68T30, 68T05, 03B70
ACM classes: I.2.3; I.2.4; I.2.6; I.2.7
Cite as: arXiv:2606.18557 [cs.AI]
(or arXiv:2606.18557v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.18557
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Patrick Cooper [view email] [v1] Wed, 17 Jun 2026 00:13:40 UTC (272 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models, by Patrick Cooper and 1 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new | recent | 2026-06
Change to browse by:
cs cs.LG cs.LO
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)