Skip to content
AI News HubLIVE
More
Summary only

Zyphra Releases ZAYA1-8B: A Reasoning MoE Trained on AMD Hardware That Punches Far Above Its Weight Class

Zyphra AI has released ZAYA1-8B, a small Mixture of Experts language model with only 760 million active parameters, outperforming models many times its size on math and coding benchmarks. The model uses the MoE++ architecture with innovations like compressed convolutional attention, MLP-based router, and learned residual scaling, along with a novel test-time compute method called Markovian RSA. Entirely trained on AMD Instinct MI300 hardware, ZAYA1-8B is available under Apache 2.0 license.

SourceMarkTechPostAuthor: Asif Razzaq
Zyphra Releases ZAYA1-8B: A Reasoning MoE Trained on AMD Hardware That Punches Far Above Its Weight Class
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions

Summary only

Under the source content permission, this page provides an editorial summary, key points, and a link to the original article.

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • ZAYA1-8B has 760M active parameters (8.4B total) and surpasses larger open-weight models in math and coding tasks.
  • MoE++ architecture includes 8× KV-cache compression via CCA, MLP-router with PID bias balancing, and learned residual scaling.
  • Novel Markovian RSA test-time compute method combines recursive self-aggregation with Markovian chunking for efficient scaling.
  • First MoE model fully trained on AMD Instinct MI300 hardware, released under Apache 2.0 license.

Highlights and analysis are generated automatically and may contain errors. Check the original source.

More recent stories on this topic · Models