Deep Pre-Alignment for VLMs
This paper proposes Deep Pre-Alignment (DPA), a novel architecture that replaces the standard ViT encoder with a small VLM as perceiver, ensuring visual features are deeply aligned with the text space of the target large language model. On the 4B parameter scale, DPA outperforms baselines by 1.9 points across 8 multimodal benchmarks, with gains widening to 3.0 points at the 32B scale. DPA also achieves a 32.9% reduction in language capability forgetting and demonstrates consistent gains across different LLM families like Qwen3 and LLaMA 3.2.
Article intelligence
Key points
- DPA replaces ViT encoder with a small VLM perceiver for deep visual-text alignment.
- Outperforms baselines by 1.9 points at 4B scale and 3.0 points at 32B scale on multimodal benchmarks.
- Reduces language capability forgetting by 32.9% and works across LLM families like Qwen3 and LLaMA 3.2.
- Offers a seamless upgrade path with modular replacement of the visual encoder and minimal computation overhead.
Why it matters
This matters because DPA replaces ViT encoder with a small VLM perceiver for deep visual-text alignment.
Technical impact
May affect model selection, inference cost, product capability, and evaluation benchmarks.
[2605.15300] Deep Pre-Alignment for VLMs
[Submitted on 14 May 2026]
Title:Deep Pre-Alignment for VLMs
View a PDF of the paper titled Deep Pre-Alignment for VLMs, by Tianyu Yu and 7 other authors
View PDF HTML (experimental)
Abstract:Most Vision Language Models (VLMs) directly map outputs from ViT encoders to the LLM via a lightweight projector. While effective, recent analysis suggests this architecture suffers from an alignment challenge: visual features remain distant from the text space in the initial layers of the LLM, forcing the model to waste critical depth~\cite{zhang-etal-2024-investigating,artzy-schwartz-2024-attend} on superficial modality alignment rather than deep understanding and complex reasoning. In this work, we propose Deep Pre-Alignment (DPA), a novel architecture that replaces the standard ViT encoder with a small VLM as perceiver, ensuring visual features are deeply aligned with the text space of the target large language model. Comprehensive experiments demonstrate the effectiveness of DPA. On the 4B parameter scale, DPA outperforms baselines by 1.9 points across 8 multimodal benchmarks, with gains widening to 3.0 points at the 32B scale. Moreover, by offloading alignment to the perceiver, DPA achieves a 32.9\% reduction in language capability forgetting over 3 text benchmarks. We further demonstrate that these gains are consistent across different LLM families including Qwen3 and LLaMA 3.2, highlighting the generality of our approach. Beyond performance, DPA also offers a seamless upgrade path for current VLM development, requiring only a modular replacement for the visual encoder with marginal computation overhead.
Comments: Accepted by ICML 2026. Project Website: this https URL
Subjects:
Computer Vision and Pattern Recognition (cs.CV)
Cite as: arXiv:2605.15300 [cs.CV]
(or arXiv:2605.15300v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2605.15300
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Tianyu Yu [view email] [v1] Thu, 14 May 2026 18:14:15 UTC (1,382 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Deep Pre-Alignment for VLMs, by Tianyu Yu and 7 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CV
new | recent | 2026-05
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)