2026-06-01原文2 min readUpdated: 2026-06-01

NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models

NumLeak is a measurement framework to detect memorization of public numeric benchmarks in foundation models. Top LLMs recall exact values with high Pearson r (0.97-0.99) on financial and economic data, but performance collapses on recent holdouts. White-box logprob ranking detects memorization better than open-ended generation, and a simple system prompt defense blocks 99.8% of suffix attacks.

SourcearXiv Machine LearningAuthor: Anany Kotawala

[2605.30393] NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models

[Submitted on 28 May 2026]

Title:NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models

View a PDF of the paper titled NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models, by Anany Kotawala

View PDF HTML (experimental)

Abstract:Public numeric benchmarks appear in pretraining, so an evaluation that conditions on a date may be measuring memorized recall rather than out-of-sample skill. We introduce NumLeak, a measurement framework that combines API-boundary probes on production models with a white-box controlled validation on an open causal LM. Top-tier frontier LLMs recall the Fama-French market excess return at 3-seed pooled Pearson r=0.97-0.99 while staying within 0.15 within-25bps on the five sibling factors; comparable fidelity appears on U.S. unemployment, CPI inflation, and NOAA temperature. On a recent-release holdout, parse rate collapses to 21-57% but r stays at approximately 0.99 on months answered, the refuse-or-recall asymmetry a memorized channel predicts. The white-box experiment reproduces the dose-response, and logprob ranking detects memorization that open-ended generation misses, implying closed-API black-box probes understate the channel. A Sonnet "date to market-sentiment" regression that correlates with true Mkt-RF at r=0.74 collapses to r=0.02 once the model's own recall is residualized out. A one-line system-prompt defense blocks 99.8% of a non-adaptive single-turn suffix attack set at near-zero utility cost on conceptual and historical-narrative queries

Comments: 23 pages, 12 figures, 17 tables. Accepted at the ICML 2026 Workshop on the Impact of Memorization on Trustworthy Foundation Models (MemFM)

Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR)

Cite as: arXiv:2605.30393 [cs.LG]

(or arXiv:2605.30393v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2605.30393

arXiv-issued DOI via DataCite

Submission history

From: Anany Kotawala [view email] [v1] Thu, 28 May 2026 12:52:49 UTC (255 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models, by Anany Kotawala

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.LG

new | recent | 2026-05

Change to browse by:

cs cs.AI cs.CR

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)