Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction
This study addresses two challenges in Text-to-Sounding-Video (T2SV) generation: modal interference from shared captions and the gap between dense training captions and concise inference prompts. The authors propose a Cross-Referential Rewriter (CRR) framework to generate disentangled caption pairs.
content type paperpublished July 2026
Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction
AuthorsKaisi Guan†‡**, Xihua Wang†‡, Zhengfeng Lai, Xin Cheng†, Peng Zhang, Xiaojiang Liu, Ruihua Song†, Meng Cao
View publication
This study focuses on Text-to-Sounding-Video (T2SV) generation, which aims to generate a video with synchronized audio from text, with both modalities aligned to the text conditions. Despite progress in joint audio-video training, two critical challenges remain: (1) text conditioning is a bottleneck—shared captions (TV=TA) trigger modal interference, while a gap persists between dense training captions and concise inference user prompts, and (2) the optimal fusion mechanism for cross-modal feature interaction remains unclear. To address the first challenge, we first propose the Cross-Referential Rewriter (CRR) caption framework, a dual-agent pipeline where a Semantic Checker extracts grounded Semantic Anchors and a Cross-Modal Rewriter generates disentangled caption pairs (TV and TA), eliminating modal interference and bridging the training-inference gap.
† Renmin University of China
‡ Equal contribution
** Work done while at Apple
Revisit Large-Scale Image–Caption Data in Pre-training Multimodal Foundation Models
April 8, 2025research area Computer Vision, research area Methods and Algorithmsconference ICLR
Recent advancements in multimodal models highlight the value of rewritten captions for improving performance, yet key challenges remain. Notably, the role of synthetic captions and their interaction with original web-crawled AltTexts in pre-training is still unclear. Additionally, different multimodal foundation models may have distinct preferences for specific caption formats while the efforts of studying the optimal captions for each foundation…
Read more
Many healthcare applications are inherently multimodal, involving several physiological signals. As sensors for these signals become more common, improving machine learning methods for multimodal healthcare data is crucial. Pretraining foundation models is a promising avenue for success. However, methods for developing foundation models in healthcare are still in early exploration and it is unclear which pretraining strategies are most effective…
Read more