Skip to content
AI News HubLIVE
Summary only

NVIDIA Introduces a 4-Bit Pretraining Methodology Using NVFP4, Validated on a 12B Hybrid Mamba-Transformer at 10T Token Horizon

NVIDIA introduces a 4-bit pretraining methodology built around the NVFP4 microscaling format — combining selective BF16 layers, 16×16 Random Hadamard Transforms on Wgrad inputs, 2D weight scaling, and stochastic rounding on gradients — validated on a 12B hybrid Mamba-Transformer trained on 10 trillion tokens, the longest publicly documented 4-bit pretraining run, with downstream accuracy closely tracking the FP8 baseline (62.58% vs 62.62% on MMLU-Pro).

SourceMarkTechPostAuthor: Asif Razzaq
NVIDIA Introduces a 4-Bit Pretraining Methodology Using NVFP4, Validated on a 12B Hybrid Mamba-Transformer at 10T Token Horizon
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions

Summary only

Under the source content permission, this page provides an editorial summary, key points, and a link to the original article.

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • NVFP4 is a 4-bit microscaling format natively supported on Blackwell Tensor Cores, quantizing only linear-layer GEMMs while keeping other components in higher precision.
  • Trained a 12B hybrid Mamba-Transformer on 10T tokens, achieving 62.58% on MMLU-Pro vs 62.62% for FP8 baseline.
  • Four techniques ensure convergence: selective BF16 layers (~16%), 16×16 Random Hadamard Transforms, 2D weight block scaling, and stochastic rounding on gradients.
  • NVFP4 achieves lower loss than MXFP4 at the same token budget (1.5% vs 2.5% relative loss gap at 1T tokens) and offers up to 3× throughput over FP8 on GB300.

Highlights and analysis are generated automatically and may contain errors. Check the original source.

More recent stories on this topic · Models