Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos
Pre-trained video LLMs struggle with auxiliary streams like audio or depth maps due to modality interference. UniMVU uses instruction-aware dynamic gating at two levels (inner-modality and modality-level) to adaptively balance importance, achieving gains up to 13.5 CIDEr across six benchmarks and aligning with human-interpretable relevance.
Article intelligence
Key points
- UniMVU introduces instruction-aware gating with inner-modality gates (emphasizing salient regions) and modality-level gates (re-weighting streams), conditioned on text instructions.
- The framework combines cross-modal self-attention with instruction-driven gating modules and a fast-to-slow fusion for time-aligned streams to reduce redundancy.
- Consistent improvements over static-fusion baselines on six benchmarks (AVQA, AVSD, Music-AVQA, ScanQA, SQA3D, MVBench), with up to 13.5 CIDEr gain.
Why it matters
This matters because uniMVU introduces instruction-aware gating with inner-modality gates (emphasizing salient regions) and modality-level gates (re-weighting streams), conditioned on text instructions.
Technical impact
May affect model selection, inference cost, product capability, and evaluation benchmarks.
[2605.26232] Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos
[Submitted on 25 May 2026]
Title:Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos
View a PDF of the paper titled Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos, by Bonan Ding and 6 other authors
View PDF HTML (experimental)
Abstract:Pre-trained video large language models excel at visual reasoning. However, they struggle when videos arrive with auxiliary streams, such as audio, depth map, or dense temporal evidence. In such a scenario, uniform fusion induces modality interference, allowing irrelevant channels to distract the model. To address this issue, we present a unified multimodal video understanding framework, named UniMVU, that performs instruction-aware fusion across video, audio, depth map, or any other modality inputs via two levels of dynamic gating: inner-modality gates emphasize salient regions within each modality, whereas modality-level gates re-weight whole streams; both are conditioned on the text instruction to adaptively balance modality importance. Our UniMVU combines cross-modal self-attention with instruction-driven inner-modality gating module and a modality-level gating module with control token; for time-aligned streams we further adopt a fast-to-slow fusion scheme that reduces redundancy. Across six benchmarks (AVQA, AVSD, Music-AVQA, ScanQA, SQA3D and MVBench), our UniMVU achieves consistent gains over static-fusion baselines achieving gains as high as 13.5 in terms of CIDEr metric. Further, our analysis shows that the gating mechanism aligns with the human-interpretable modality relevance, and ablations show the contributions of inner-modality and modality-level gating. Our UniMVU provides a simple, unified recipe for instruction-aware multimodal video understanding that scales to diverse modalities without hand-crafted fusion rules.
Comments: 19 pages, 8 figures, 7 tables, preprint
Subjects:
Computer Vision and Pattern Recognition (cs.CV)
Cite as: arXiv:2605.26232 [cs.CV]
(or arXiv:2605.26232v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2605.26232
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Bonan Ding [view email] [v1] Mon, 25 May 2026 18:02:20 UTC (11,190 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos, by Bonan Ding and 6 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CV
new | recent | 2026-05
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)