Skip to content
AI News HubLIVE
Original source2 min read

When benchmark inferences do not compose: Projectibility in AI evaluation

Summary

This paper identifies an epistemic problem beyond standard validity theory: warranted inferences from AI benchmarks do not automatically compose into a warranted chain. The author proposes a non-composition principle: support for adjacent projections warrants their composition only when endpoints and assumptions align, and dependence and uncertainty are carried through. A legal-research case, a reanalysis, and a simulation show why aggregate stability can erase distinctions a later projection requires, and the paper introduces a projectibility audit to diagnose unsupported joins in benchmark-to-use arguments.

SourcearXiv AIAuthor: Brett Reynolds
When benchmark inferences do not compose: Projectibility in AI evaluation
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[Submitted on 28 Jul 2026]

Title:When benchmark inferences do not compose: Projectibility in AI evaluation

View a PDF of the paper titled When benchmark inferences do not compose: Projectibility in AI evaluation, by Brett Reynolds

View PDF HTML (experimental)

Abstract:An AI benchmark result rarely reaches a consequential claim in one step. Evaluators generalize it to further cases, interpret it as evidence of capability, extrapolate it to new tasks, transport it to another system or site, and combine it with assumptions about human review and downstream consequences. Validity-centred approaches require evidence for each claim. This paper identifies a further epistemic problem: warranted links don't automatically make a warranted chain. The target of one study may not be the source of the next; system, population, outcome, or conditions may change at the interface; and shared data or model lineage may make apparently independent support dependent. Projectibility concerns whether a bounded extension from observed to unobserved cases is warranted. Goodman supplies the problem of rival extensions; argument-based validity supplies an architecture for testing them. The paper's distinctive claim is a non-composition principle: support for adjacent projections warrants their composition only when endpoints and assumptions align and dependence and uncertainty are carried through. A legal-research case shows how benchmark evidence and a deployment study can each be sound while remaining parallel. A reanalysis and simulation show why aggregate stability can erase distinctions a later projection requires. The resulting projectibility audit diagnoses unsupported joins in benchmark-to-use arguments.

Comments: 35 pages, 2 figures, 5 tables. Substantially rewritten and retitled; supersedes arXiv:2510.15236, whose homeostatic property-cluster account and proposed centrality-prior and cluster-stability measures are withdrawn. The argument, apparatus, and empirical companion are new. Code and empirical companion: this https URL

Subjects:

Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Machine Learning (cs.LG)

Cite as: arXiv:2607.26159 [cs.AI]

(or arXiv:2607.26159v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2607.26159

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Brett Reynolds [view email] [v1] Tue, 28 Jul 2026 18:07:04 UTC (2,852 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled When benchmark inferences do not compose: Projectibility in AI evaluation, by Brett Reynolds

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.AI

new | recent | 2026-07

Change to browse by:

cs cs.CY cs.LG

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

InvestorsAdvanced

Key points

  • Warranted links do not automatically make a warranted chain because targets, populations, outcomes, or conditions may change at the interface between studies.
  • Shared data or model lineage can make ostensibly independent support dependent, undermining composition.
  • A legal-research case demonstrates that benchmark evidence and a deployment study can each be sound yet remain parallel and non-composable.
  • The proposed projectibility audit diagnoses unsupported joins in benchmark-to-use arguments.

Highlights and analysis are generated automatically and may contain errors. Check the original source.