On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs
Reinforcement learning finetuning improves vision-language models on reasoning benchmarks, but they remain vulnerable to textual perturbations. This paper shows that misleading captions or incorrect chain-of-thought traces cause significant drops in robustness, especially when consistency is considered. Closed models outperform open-source ones, indicating a gap in current open-source RL finetuning. An accuracy-faithfulness trade-off is identified; adversarial augmentation alone is insufficient, and a faithfulness-aware reward can help but may lead to shortcut learning.
content type paperpublished July 2026
On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs
AuthorsRosie Zhao†, Anshul Shah, Xiaoyu Zhu, Xinke Deng, Zhongyu Jiang, Yang Yang‡, Joerg Liebelt, Arnab Mondal
View publication
Reinforcement learning (RL) finetuning has become a key technique for enhancing large language models (LLMs) on reasoning-intensive tasks, motivating its extension to vision language models (VLMs). While RL-tuned VLMs improve on visual reasoning benchmarks, they remain vulnerable to weak visual grounding, hallucinations, and over-reliance on textual cues. We show that simple, controlled textual perturbations—misleading captions or incorrect chain-of-thought (CoT) traces—cause substantial drops in robustness and confidence, and that these effects are more pronounced when CoT consistency is taken into account across open-source multimodal reasoning models. In contrast, closed models exhibit similar failure modes but maintain markedly greater robustness and reasoning consistency, suggesting that the gap reflects a shortcoming in current open-source RL finetuning rather than an inherent limitation of the task. To better understand these vulnerabilities, we further analyze RL finetuning dynamics and uncover an accuracy–faithfulness trade-off: finetuning raises benchmark accuracy, but can simultaneously erode the reliability of the accompanying CoT and its robustness to contextual shifts. Although adversarial augmentation improves robustness, it does not by itself prevent faithfulness drift. Incorporating a faithfulness-aware reward can restore alignment between answers and reasoning, but when paired with augmentation, training risks collapsing onto shortcut strategies and robustness remains elusive. Together, these findings highlight the limitations of accuracy-only evaluations and motivate training and assessment protocols that jointly emphasize correctness, robustness, and the faithfulness of visually grounded reasoning.
† Harvard University
‡ OpenAI
** Work done while at Apple
The Potential of CoT for Reasoning: A Closer Look at Trace Dynamics
February 24, 2026research area Methods and Algorithms, research area Speech and Natural Language Processingconference ICLR
Chain-of-thought (CoT) prompting is a de-facto standard technique to elicit reasoning-like responses from large language models (LLMs), allowing them to spell out individual steps before giving a final answer. While the resemblance to human-like reasoning is undeniable, the driving forces underpinning the success of CoT reasoning still remain largely unclear. In this work, we perform an in-depth analysis of CoT traces originating from…
Read more
Improve Vision Language Model Chain-of-thought Reasoning
June 5, 2025research area Computer Vision, research area Speech and Natural Language Processingconference ACL
Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial for improving interpretability and trustworthiness. However, current training recipes often relying on datasets dominated by short annotations with minimal rationales. In this work, we show that training VLM on short answers leads to poor generalization on reasoning tasks that require more detailed explanations. To address this limitation, we propose a two-stage…
Read more