2026-06-03原文2 min readUpdated: 2026-06-03

Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection

Research reveals that within-dataset class-split evaluation for anomaly detection can become ill-posed when the held-out anomaly class overlaps the normal mixture in representation space. Scores may collapse to chance or invert, and preferred direction depends on unknown anomaly class. A training-free diagnostic, neighborhood class leakage, is introduced and shown to predict instability across multiple datasets and latent spaces. The paper argues that class-split benchmarks should be treated as geometry-dependent stress tests rather than unconditional evidence of detection ability.

SourcearXiv Machine LearningAuthor: Alejandro Ascarate, Leo Lebrat, Rodrigo Santa Cruz, Clinton Fookes, Olivier Salvado

[2606.02601] Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection

[Submitted on 23 May 2026]

Title:Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection

View a PDF of the paper titled Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection, by Alejandro Ascarate and 4 other authors

View PDF HTML (experimental)

Abstract:Within-dataset class-split evaluation is widely used as a proxy for fully unconditional out-of-distribution anomaly detection. We show that this protocol can become ill-posed when the held-out anomaly class overlaps the normal mixture in representation space. In this regime, anomaly scores may collapse toward chance or even invert, and the preferred score direction can depend on the unknown anomaly class. We introduce a simple training-free diagnostic, neighborhood class leakage, and show that it predicts score-direction instability across Fashion-MNIST, CIFAR-10, and Imagenette, in both pixel and VAE latent spaces. Our results suggest that class-split AD benchmarks should be treated as geometry-dependent stress tests rather than unconditional evidence of anomaly-detection ability.

Comments: 4+1 pages, 1 figure, accepted at ICML 2026 Workshop on Hypothesis Testing

Subjects:

Machine Learning (cs.LG)

Cite as: arXiv:2606.02601 [cs.LG]

(or arXiv:2606.02601v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2606.02601

arXiv-issued DOI via DataCite

Submission history

From: Alejandro Ascárate [view email] [v1] Sat, 23 May 2026 06:27:42 UTC (114 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection, by Alejandro Ascarate and 4 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.LG

new | recent | 2026-06

Change to browse by:

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)