Residual Context Diffusion Language Models
Residual Context Diffusion (RCD) is a new module for diffusion large language models (dLLMs) that recycles computation from discarded tokens, improving efficiency and accuracy. RCD converts discarded token representations into contextual residuals and reinjects them into the denoising process. It uses a decoupled two-stage training pipeline and achieves 5-10 point accuracy gains across benchmarks, with up to 4-5x fewer denoising steps on challenging tasks like AIME.
content type paperpublished July 2026
Residual Context Diffusion Language Models
AuthorsYuezhou Hu†*, Harman Singh†*, Monishwaran Maheswaran†*, Haocheng Xi†, Coleman Hooper†, Jintao Zhang†, Aditya Tomar†, Michael W. Mahoney†, Sewon Min†, Mehrdad Farajtabar, Kurt Keutzer†, Amir Gholami†‡, Chenfeng Xu†‡
View publication
Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to purely autoregressive language models because they can decode multiple tokens in parallel. However, state-of-the-art block-wise dLLMs rely on a “remasking” mechanism that decodes only the most confident tokens and discards the rest, effectively wasting computation. We demonstrate that recycling computation from the discarded tokens is beneficial, as these tokens retain contextual information useful for subsequent decoding iterations. In light of this, we propose Residual Context Diffusion (RCD), a module that converts these discarded token representations into contextual residuals and injects them back for the next denoising step. RCD uses a decoupled two-stage training pipeline to bypass the memory bottlenecks associated with backpropagation. We validate our method on both long CoT reasoning (SDAR) and short CoT instruction following (LLaDA) models. We demonstrate that a standard dLLM can be efficiently converted to the RCD paradigm with merely ∼1 billion tokens. RCD consistently improves frontier dLLMs by 5–10 points in accuracy with minimal extra computation overhead across a wide range of benchmarks. Notably, on the most challenging AIME tasks, RCD nearly doubles baseline accuracy and attains up to 4–5x fewer denoising steps at equivalent accuracy levels.
† University of California, Berkeley
- Equal contribution
‡ Equal advising
Learning Unmasking Policies for Diffusion Language Models
July 2, 2026research area Methods and Algorithms, research area Speech and Natural Language Processingconference ICML
Diffusion (Large) Language Models (dLLMs) now match the downstream performance of their autoregressive counterparts on many tasks, while holding the promise of being more efficient during inference. One critical design aspect of dLLMs is the sampling procedure that selects which tokens to unmask at each diffusion step. Indeed, recent work has found that heuristic strategies such as confidence thresholding improve both sample quality and token…
Read more
DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation
January 21, 2026research area Speech and Natural Language Processingconference ICLR
Diffusion large language models (dLLMs) are compelling alternatives to autoregressive (AR) models because their denoising models operate over the entire sequence. The global planning and iterative refinement features of dLLMs are particularly useful for code generation. However, current training and inference mechanisms for dLLMs in coding are still under-explored. To demystify the decoding behavior of dLLMs and unlock their potential for coding,…
Read more