Skip to content
AI News HubLIVE
More
Summary only

He Kaiming's First Language Model: 105M Parameters, Breaking Away from GPT's Autoregressive Path

He Kaiming's team introduces ELF, a continuous diffusion language model with only 105M parameters, 45B training tokens, and 32 sampling steps, achieving a generative perplexity of 24 on OpenWebText, surpassing mainstream diffusion LMs. ELF performs denoising entirely in continuous embedding space and discretizes only at the final step, eliminating the need for an extra decoder.

Source量子位Author: henry
He Kaiming's First Language Model: 105M Parameters, Breaking Away from GPT's Autoregressive Path
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions

Summary only

Under the source content permission, this page provides an editorial summary, key points, and a link to the original article.

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • He Kaiming's team releases first language model ELF, adopting continuous diffusion instead of autoregression.
  • 105M parameters, 45B training tokens, 32 steps, generative perplexity 24.
  • ELF denoises in continuous embedding space and discretizes at the last step, achieving high-quality generation.
  • Outperforms existing diffusion models and some autoregressive baselines on translation and summarization.

Highlights and analysis are generated automatically and may contain errors. Check the original source.

More recent stories on this topic · Models