Skip to content
AI News HubLIVE
More
Summary only

Zyphra Releases ZAYA1-8B-Diffusion-Preview: The First MoE Diffusion Model Converted From an Autoregressive LLM With Up to 7.7x Speedup

Zyphra's latest release shows that an autoregressive MoE model can be converted into a discrete diffusion model with no systematic loss in evaluation performance. ZAYA1-8B-Diffusion-Preview achieves up to 7.7x inference speedup over autoregression by shifting decoding from memory-bandwidth bound to compute-bound — a key advantage as modern GPUs continue scaling FLOPs faster than memory bandwidth.

SourceMarkTechPostAuthor: Asif Razzaq
Zyphra Releases ZAYA1-8B-Diffusion-Preview: The First MoE Diffusion Model Converted From an Autoregressive LLM With Up to 7.7x Speedup
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions

Summary only

Under the source content permission, this page provides an editorial summary, key points, and a link to the original article.

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • Zyphra releases ZAYA1-8B-Diffusion-Preview, the first MoE diffusion model converted from an autoregressive LLM.
  • It achieves up to 7.7x inference speedup by generating 16 tokens simultaneously with shared KV cache, converting decoding from memory-bandwidth bound to compute-bound.
  • Offers two samplers: lossless (4.6x speedup) and logit-mixing (7.7x speedup) with runtime speed-quality trade-off.
  • Model is based on ZAYA1-8B, trained with 1.1 trillion tokens of diffusion conversion and context extension, run on AMD GPUs.

Highlights and analysis are generated automatically and may contain errors. Check the original source.

More recent stories on this topic · Models