Scaling Properties of Continuous Diffusion Spoken Language Models
This study investigates the scaling properties of continuous diffusion (CD) spoken language models (SLMs), introducing the phoneme Jensen-Shannon divergence (pJSD) metric. CD SLMs exhibit scaling laws similar to autoregressive models, with loss flattening at higher compute allowing flexible parameter-data allocation. Scaling to 16B parameters enables emotive, multi-speaker, multilingual speech, but long-form coherence remains challenging.
content type paperpublished July 2026
Scaling Properties of Continuous Diffusion Spoken Language Models
AuthorsJason Ramapuram†‡, Eeshan Gunesh Dhekane‡, Amitis Shidani‡, Dan Busbridge, Bogdan Mazoure†, Zijin Gu, Russ Webb, Tatiana Likhomanenko§, Navdeep Jaitly†§**
View publication
Speech-only spoken language models (SLMs) lag behind text and text-speech models in performance, with recent discrete autoregressive (AR) SLMs indicating significant computational and data demands to match text models. Since discretizing continuous speech for AR creates bottlenecks, we explore whether continuous diffusion (CD) SLM is more viable. To quantify the SLMs linguistic quality, we introduce the phoneme Jensen-Shannon divergence (pJSD) metric. Our analysis reveals CD SLMs, mirroring AR behavior, exhibit scaling laws for validation loss and pJSD, and show optimal token-to-parameter ratios decreasing as compute scales. However, for the latter, loss becomes insensitive to choice of data and model sizes, showing potential for fast inference. Scaling CD SLMs to 16B parameters with tens of millions of hours of conversational data enables generation of emotive, prosodic, multi-speaker, multilingual speech, though achieving long-form coherence remains a significant challenge.
‡ Core contributor
§ Core advising
** Work done while at Apple
Figure 1: Scaling law fit for validation loss. Training (•) and testing (×) points are shown alongside compute-optimal points (★).
Figure 2: The curvature κ of isoFLOPs at their optima decreases as compute increases: flattening corresponds to approximately two orders of magnitude expansion in the range of model (ΔN) and dataset (ΔD) sizes yielding a loss within ε of the optimum L*. Thus, higher compute allows near-optimal performance across a much wider variety of parameter-to-data allocations, opening up an efficient inference frontier.
LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Loss
April 9, 2026research area Methods and AlgorithmsWorkshop at ICLR
This paper was accepted at the Workshop on Memory for LLM-Based Agentic Systems at ICLR.
Language models have consistently grown to compress more world knowledge into their parameters, but the knowledge that can be pretrained into them is upper-bounded by their parameter size. Especially the capacity of Small Language Models (SLMs) is limited, leading to factually incorrect generations. This problem is often mitigated by giving the SLM access…
Read more
Computational Bottlenecks of Training Small-Scale Large Language Models
October 29, 2024research area Methods and Algorithms, research area Speech and Natural Language ProcessingWorkshop at NeurIPS
This paper was accepted at the Efficient Natural Language and Speech Processing (ENLSP) Workshop at NeurIPS 2024.
While large language models (LLMs) dominate the AI landscape, Small-scale large Language Models (SLMs) are gaining attention due to cost and efficiency demands from consumers. However, there is limited research on the training behavior and computational requirements of SLMs. In this study, we explore the computational bottlenecks of…
Read more