2026-05-25原文

Learnability-Informed Fine-Tuning of Diffusion Language Models

To improve reasoning in diffusion language models (DLMs), researchers propose LIFT, a fine-tuning algorithm that adapts to token learnability across diffusion steps, outperforming baselines on six benchmarks with up to 3x relative gains on AIME'24 and AIME'25.

Article intelligence

EngineersAdvanced

Key points

Standard SFT overlooks learnability, potentially harming DLM performance.
LIFT learns easy tokens when masked and hard tokens when context is available.
Achieves up to 3x relative improvement on AIME'24 and AIME'25.

Why it matters

This matters because standard SFT overlooks learnability, potentially harming DLM performance.

Technical impact

May affect model selection, inference cost, product capability, and evaluation benchmarks.

[2605.22939] Learnability-Informed Fine-Tuning of Diffusion Language Models

[Submitted on 21 May 2026]

Title:Learnability-Informed Fine-Tuning of Diffusion Language Models

View a PDF of the paper titled Learnability-Informed Fine-Tuning of Diffusion Language Models, by Shubham Parashar and 7 other authors

View PDF HTML (experimental)

Abstract:We aim to improve the reasoning capabilities of diffusion language models (DLMs). While SFT is a popular post-training recipe for autoregressive models, its use in DLMs faces challenges and can even hurt performance, though the underlying causes remain understudied. Our analysis reveals that vanilla SFT overlooks learnability, namely what and when tokens are learned. Specifically, rare tokens are difficult to learn when most of the input is masked, whereas it is straightforward and thus of little value to learn common tokens when most of the input is unmasked. Motivated by our analysis, we propose LIFT, an efficient SFT-based post-training algorithm for DLMs. LIFT learns easy tokens when most of the input is masked and hard tokens when more context is available, thus aligning the training with the information available at different diffusion time steps. Our results show that LIFT outperforms existing SFT baselines across six reasoning benchmarks, achieving up to a 3x relative gain on AIME'24 and AIME'25. Our code is publicly available at this https URL.

Subjects:

Computation and Language (cs.CL); Machine Learning (cs.LG)

Cite as: arXiv:2605.22939 [cs.CL]

(or arXiv:2605.22939v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2605.22939

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Shubham Parashar [view email] [v1] Thu, 21 May 2026 18:16:17 UTC (3,786 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Learnability-Informed Fine-Tuning of Diffusion Language Models, by Shubham Parashar and 7 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-05

Change to browse by:

cs cs.LG

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)