待翻译：Continuous Diffusion Language Models (CDLM's)

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

A flurry of recent activity in the space of continuous diffusion models for language, after a few years of relative dormancy, suggests that this approach is making something of a comeback. Fully discrete diffusion methods had largely supplanted earlier attempts to make continuous diffusion work for language, but the tide is starting to turn. In this post, I want to take a closer look at what’s going on, and why it is happening now. The recent influx of new research in this space inspired me to write up some of my thoughts. I have written about diffusion language models before, so this mainly serves as an update to cover everything that’s happened since then. This will be a fairly subjective account – other perspectives and dissenting opinions are very welcome in the comments and elsewhere! I’ll discuss some technical aspects of continuous diffusion for language later on, but first, some historical context. Challenging the autoregressive hegemony Modern language models are, by and large, autoregressive: they generate sequences one token at a time. This is a natural decomposition of a difficult generation task into smaller, easier sequential steps. All steps are instances of the same underlying task (predict a token given preceding tokens), which enables parameter sharing across the sequence dimension. In spite of this inherently sequential generative process, the Transformer architecture1 admits efficient parallel training across all sequence positions using teacher forcing2. This has turned out to be an extremely scalable recipe3, which has brought us large language models (LLMs). However, autoregression is not the only way to construct an iterative generative process for sequences. Inspired by early successes in the audiovisual domain, researchers sought to apply diffusion to language generation instead. Rather than generating a sequence one element at a time, the generative process of diffusion models is defined by reversing a corruption process, which gradually destroys information. The canonical way to do this is to add Gaussian noise little by little, until it completely overpowers the signal. 2021: early discrete diffusion models After early successes in image generation in 20194 and 20205 6, the first attempts to apply this idea to language arrived in 2021, and involved replacing a continuous corruption process with a discrete one to enable modelling of categorical data: multinomial diffusion7, D3PM8 and SUNDAE9. Back then, the dominance of autoregression was not as well-established as it is today: GPT-310 had turned some heads, but the ‘ChatGPT moment’ wouldn’t come until late 2022. At the time, discrete diffusion seemed to address some real theoretical flaws in the autoregressive paradigm, like exposure bias due to teacher forcing and the relative difficulty of applying it to infilling and constrained generation tasks. Note that there had been some exploration of non-autoregressive and any-order autoregressive approaches in the preceding years11 12 (especially for machine translation13 14), but not yet from a diffusion perspective. 2022: continuous diffusion for discrete data In 2022, several attempts to apply continuous diffusion to language modelling appeared, starting with Diffusion-LM15. This approach addresses the incompatibility between categorical data and corruption with Gaussian noise in a different way: simply represent the discrete categories with continuous embedding vectors, which are perfectly amenable to Gaussian noise corruption. That way, the Gaussian diffusion mechanism, which works so well for images, can be applied without any changes. Diffusion-LM touted the advantages of this alternative generative paradigm for controllable text generation in particular. In the last few months of 2022, quite a few other papers using variations of this approach were published, including DiffuSeq16, SSD-LM17, Difformer18, SeqDiffuSeq19, GENIE20, LD4LG21 and also two papers that I worked on: self-conditioned embedding diffusion (SED)22 and continuous diffusion for categorical data (CDCD)23. At the time, the allure of these continuous methods was that they could benefit from all the insights, tools and machinery that were being discovered and developed for continuous diffusion, as it completely took over audiovisual generation. For example, applying some of the sampling and distillation techniques developed for continuous diffusion models to discrete diffusion was often much less straightforward, or even downright impossible. Late 2023: the continuous extinction Then, something interesting happened: after 2023, virtually all new research in this space used discrete diffusion, and continuous diffusion for language went extinct. A diagram from a 2025 survey paper24 about diffusion language models clearly shows this: New survey on diffusion language models: https://t.co/SHicf69gxV (via @NicolasPerezNi1). Covers pre/post-training, inference and multimodality, with very nice illustrations. I can't help but feel a bit wistful about the apparent extinction of the continuous approach after 2023🥲 pic.twitter.com/RYvLHuLHWH — Sander Dieleman (@sedielem) August 19, 2025 Continuous methods are marked in yellow, discrete methods in green. The transition from 2023 to 2024 is quite stark! It is difficult to say for certain why this happened, but I can think of a few potential factors: one is the ChatGPT moment, which gradually shifted the focus of language diffusion research from theoretical advantages and elegance to raw performance. Now, the goal was to try and match powerful autoregressive models at scale, or even outcompete them in specific settings. It seems that people felt closing the performance gap would be easier to achieve with fully discrete methods, perhaps because they are conceptually more closely related to autoregression. Another factor could be that the science of scaling language diffusion models had started to be explored, and initial observations for continuous methods weren’t looking promising. In May 2023, Gulrajani & Hashimoto25 quantified the training efficiency gap for a likelihood-based continuous diffusion language model (Plaid-1B): 64x less efficient. (Note that the diagram above marks Plaid as discrete, but it is a continuous method.) At a time when the LLM community was still very much focused on the pareto frontier of training compute versus perplexity (Chinchilla-optimality26), any modelling approach whose training efficiency was almost two orders of magnitude worse than an autoregressive baseline was difficult to take seriously. The first LLaMA27 model, which challenged this training efficiency focus and argued for taking the inference budget into account, had only just been released a few months earlier (February 2023), so I believe it is plausible that the community had not yet internalised this shift. Needless to say, this is all highly speculative. Perhaps it was just a coincidence, and discrete methods ended up having more momentum around that time purely by chance. If you have any thoughts about what could have caused the late-2023 continuous extinction event, I’d be keen to hear them in the comments! Personally, I had stopped working on diffusion language models by that point (I got too busy building image and video generation models: Imagen and Veo, and later on, Nano Banana and Omni), so I was just observing this evolution from the sidelines. I found it somewhat suprising, because I believed continuous diffusion has a few key advantages, like an ability to represent uncertainty at the individual token level, and a rich toolbox of sampling algorithms and tricks to draw on. Giving those up seemed like it could be a mistake, but the research community as a whole clearly figured that this was the way to go. Adapting continuous diffusion to discrete data We will talk about what’s been happening in the diffusion language modelling space more recently in the next section, but first, I think it is useful to discuss how continuous diffusion can actually be applied to discrete data in a bit more detail. This context will be helpful to understand what might be driving recent events. The first thing to consider is the nature of the discrete data we are trying to model. Usually when people say ‘discrete’, they actually mean categorical, i.e. the output space (at the token level) is an unstructured set, and there is no relationship (ordinal or otherwise) between the different values that each discrete variable can assume. Digital images represented as pixel grids are also discrete, technically speaking, but because the discrete values assumed by the pixel colour channels represent an underlying continuous physical signal (light intensity), we tend to simply ignore that, and treat them as continuous anyway. Assuming we are working with categorical data, there are a few necessary ingredients to make continuous diffusion work well: an embedding strategy, a loss function, and a sensible noise schedule. In addition, there is a trick that pops up in almost every paper on this topic, which turns out to have a huge impact on performance: self-conditioning. We’ll take a closer look at each of these in turn. For brevity, I will use the acronym CDLM to refer to continuous diffusion language models going forward, and DDLM to refer to their discrete counterparts. Embedding strategies 📍 Modern neural networks typically have real-valued parameters and activations. Therefore, the first thing that usually happens in any neural network that processes discrete data, is embedding the discrete inputs in a continuous representation space. From that point on, the network exclusively manipulates real-valued representations. These embeddings are usually just treated as additional parameters, which can be optimised jointly with the rest of the model. It’s worth pointing out that this is also the case for DDLMs and autoregressive models – the internals of the neural networks powering these models are still continuous. To apply continuous diffusion to discrete data, we can simply ‘lift’ the corruption process from the discrete input space into a continuous embedding space. In other words, compared to discrete diffusion, it’s just a question of changing the order of operations. Rather than applying discrete corruption followed by continuous embedding during training, we first embed the inputs and then apply continuous corruption instead. Schematic diagram of (a) an autoregressive model, (b) a discrete diffusion model (masked diffusion) and (c) a continuous diffusion model. Green blocks indicate continuous embeddings of the discrete tokens. Blue blocks represent the layers of the model. For a and b, the embedding stage is typically considered also part of the model. Corruption for b happens at the token level, before embedding. For c, corruption is applied by adding noise to the continuous embeddings. The shape and structure of the embedding space profoundly impacts the nature of the continuous corruption process that happens within it. Various embedding strategies have been explored: Explicit (e.g. SSD-LM17): arguably the simplest approach is to use something like a one-hot representation, where every element in a vocabulary of size \(V\) is associated with a \(V\)-dimensional vector that has \(V-1\) zeros and a single one. Since a vocabulary is a set, assigning representations to the elements requires arbitrarily picking a specific ordering. This kind of embedding space can be cumbersome to work with for modern language models, because \(V\) tends to be pretty large nowadays. A potential workaround is to use compact binary patterns instead, as in Analog Bits28. Pre-trained (e.g. SED22): we can use a representation learning strategy to learn embeddings, and then repurpose them for use in a diffusion language model. For example, they could be borrowed from an autoregressive language model, or taken from [truncated for AI cost control]