2026-05-21 04:00 UTCOriginal source2 min readUpdated: 2026-06-30 13:03 UTC

Data Scaling as Progressive Coverage of a Predictive Contribution Spectrum

This paper proposes that data scaling laws are governed by progressive coverage of a latent predictive contribution spectrum, not just token-frequency tails. Using a suffix-automaton representation, the authors define a global-KL spectrum whose tail slope correlates strongly with the empirical scaling exponent of a small GPT model across 12 corpora. They further define an effective truncation rank K(N) that scales linearly in log-log space with training size N, supporting a simple mechanism where training scale advances through a state spectrum.

SourcearXiv Computational LinguisticsAuthor: Zihui Song, Shihao Ji, Hongxi Li, Shuaizhi Cheng, Chunlin Huang

[2605.20196] Data Scaling as Progressive Coverage of a Predictive Contribution Spectrum

[Submitted on 5 Apr 2026]

Title:Data Scaling as Progressive Coverage of a Predictive Contribution Spectrum

View a PDF of the paper titled Data Scaling as Progressive Coverage of a Predictive Contribution Spectrum, by Zihui Song and 4 other authors

View PDF HTML (experimental)

Abstract:We investigate the hypothesis that real-data scaling laws are governed by progressive coverage of a latent predictive contribution spectrum rather than by token-frequency tails alone. We work with a suffix-automaton representation of text corpora and define a data-intrinsic global-KL predictive contribution spectrum, in which each state contributes according to its empirical mass times its KL deviation from a global next-token baseline. Across 12 real corpora, the tail slope of this spectrum is already strongly correlated with the empirical data-scaling exponent of a fixed small GPT learner. We then go beyond slope correlation and define, for each training size N, an effective truncation rank K(N) by matching the observed excess loss to the residual tail mass of the prepared 1000k global-KL spectrum. Empirically, log K is close to linear in log N, with pooled R^2 about 0.96 for the raw spectrum and R^2 about 0.90 for the smoothed spectrum. These findings provide strong empirical support for a simple mechanism picture: training scale advances an effective frontier through a predictive state spectrum, and the residual tail mass of that spectrum tracks the remaining excess loss.

Comments: 8 pages,6 figures

Subjects:

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

Cite as: arXiv:2605.20196 [cs.CL]

(or arXiv:2605.20196v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2605.20196

arXiv-issued DOI via DataCite

Submission history

From: HongXi Li [view email] [v1] Sun, 5 Apr 2026 10:47:24 UTC (1,382 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Data Scaling as Progressive Coverage of a Predictive Contribution Spectrum, by Zihui Song and 4 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-05

Change to browse by:

cs cs.AI cs.LG

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)