Don't Rush from SFT to RL! Your Multimodal LLM May Be Training 'Injured'
A study reveals that the common SFT→RL pipeline for multimodal LLMs causes performance degradation after SFT due to distributional drift, forcing RL to merely recover lost ground. PRISM introduces an intermediate distribution alignment stage with a mixture-of-experts discriminator to separately correct perceptual and reasoning drifts, achieving significant gains across benchmarks.
A team of researchers from HKUST(GZ), Nanyang Technological University, and Tsinghua University has identified a critical flaw in the conventional post-training pipeline for multimodal large language models (MLLMs). The widely adopted two-stage paradigm — supervised fine-tuning (SFT) followed by reinforcement learning (RL) — may actually be hurting model performance before RL even begins. Their solution, called PRISM, introduces a third stage between SFT and RL to repair the damage caused by SFT.
The problem stems from distributional drift induced by SFT. When SFT is applied to already strong instruction-tuned models (e.g., Qwen3-VL), the fine-tuning data often comes from a different distribution — typically distilled outputs from GPT or Gemini. This forces the model to mimic a narrower distribution, overwriting its broader capabilities without truly learning new ones. The result is a significant performance drop: on an average of seven mainstream multimodal benchmarks, Qwen3-VL-4B dropped 3.0% after SFT, and Qwen3-VL-8B dropped 5.2%. Subsequent RL (e.g., GRPO) only brought the 8B model back to its original baseline (63.3%), meaning the RL training merely repaid the debt incurred by SFT rather than delivering actual improvement.
The authors identify two specific types of bias introduced by SFT in multimodal settings. First, token-level loss treats correct reasoning steps and formatting templates equally, leading to superficial imitation — the model learns to look correct without genuine understanding. Second, SFT conflates two qualitatively different failure modes: perceptual drift (the model misinterprets visual input) and reasoning drift (the model makes logical errors). Both are fitted with the same loss, making them difficult to correct separately.
Existing RL algorithms, from GRPO to DAPO to GSPO, focus on internal issues like sampling efficiency and gradient variance. None of them address the distributional misalignment left by SFT. As the authors colorfully put it: SFT pushes the runner 50 meters backward, and all RL algorithms try to make the runner faster — but from a starting point still in the hole. PRISM aims to pull the model back to the starting line and even give it a push forward.
PRISM proposes a three-stage pipeline: SFT → distribution alignment (PRISM) → RLVR. The core innovation is the alignment stage, which employs a mixture-of-experts (MoE) discriminator with two specialized experts: a perceptual expert D_v that evaluates whether outputs faithfully reflect image content, and a reasoning expert D_r that assesses logical consistency. The final reward is a weighted combination of the two scores, providing decoupled correction signals. This avoids the noise that would arise from packing both error types into a single scalar.
Another elegant design choice is black-box distillation: PRISM works entirely at the response level, sampling high-quality outputs from a strong model (Gemini 3 Flash) as positive examples and current policy outputs as negatives. It requires no access to teacher logits — just an API call — making it practical for real-world scenarios where the strongest models are only available via APIs.
Notably, PRISM deliberately omits KL regularization. Since the goal is to move the policy away from the SFT-induced distribution, adding a KL penalty would counteract that objective.
Extensive experiments on Qwen3-VL models (4B and 8B) combined with GRPO, DAPO, and GSPO show consistent improvements. On average, PRISM boosts Qwen3-VL-4B by +4.4% and Qwen3-VL-8B by +6.0% across four math reasoning and three general multimodal benchmarks. The stronger the base model, the greater the gain, because stronger models are more adversely affected by SFT drift. Ablation studies confirm that removing the alignment stage causes a 4.4% drop, and replacing the MoE discriminator with a single 4B discriminator reduces gains by 3.4%, highlighting the importance of the decoupled design.
The research is available on arXiv (https://arxiv.org/abs/2604.28123) and code is open-sourced on GitHub (https://github.com/XIAO4579/PRISM). The team believes that fixing the distributional gap between SFT and RL is a more effective path to advancing MLLMs than developing ever more complex RL algorithms or collecting more data.