Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?
This paper proposes the Ablate-to-Validate diagnostic principle and its instantiation, the Token Replacement Test (TRT), to determine whether vision-language models (VLMs) genuinely use continuous latent tokens for reasoning. Experiments show that VLMs retain most performance gains even when token content is corrupted or replaced, indicating that accuracy improvements are a misleading proxy for latent-token reasoning.
[2605.21642] Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?
[Submitted on 20 May 2026]
Title:Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?
View a PDF of the paper titled Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?, by Tianyi Zhang and 2 other authors
View PDF HTML (experimental)
Abstract:Vision-language models (VLMs) are increasingly augmented with continuous or latent non-textual tokens intended to support "visual thinking." Despite improved task accuracy, this alone does not show that models actually use these tokens for reasoning -- gains may arise from confounds such as added context length, special-token anchoring, or training-time regularization. We formalize a diagnostic principle, Ablate-to-Validate, for testing whether latent-token content is genuinely utilized, and instantiate it as the Token Replacement Test (TRT), a standardized suite of content-replacement ablations. TRT holds the prompt, image, token budget, and decoding fixed while replacing intermediate tokens with zero, random, first-repeat, or oracle alternatives, isolating whether performance depends on token content or merely on token presence. As a controlled testbed, we study relative depth reasoning with LLaVA-13B and Qwen2.5-VL-3B, training models to predict and consume continuous or discrete depth spans across multiple frozen encoders (SigLIP2, CLIP, DINOv2) and token budgets. We additionally apply TRT to three off-the-shelf visual-thinking systems (Mirage, Mull-Tokens, CoVT) on BLINK, VSP, and CV-Bench. Across all settings, accuracy gains are a misleading proxy for latent-token reasoning: VLMs retain most improvement even when token content is corrupted or replaced, revealing a persistent gap between having a latent channel and using it as an information bottleneck. We recommend TRT as a standard diagnostic alongside accuracy for any method introducing continuous thought tokens.
Subjects:
Computer Vision and Pattern Recognition (cs.CV)
Cite as: arXiv:2605.21642 [cs.CV]
(or arXiv:2605.21642v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2605.21642
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Tianyi Zhang [view email] [v1] Wed, 20 May 2026 18:55:16 UTC (96 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?, by Tianyi Zhang and 2 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CV
new | recent | 2026-05
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)