待翻譯：Nemotron 3 Ultra Explained

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

Back to blog list Nemotron 3 Ultra Explained: NVIDIA's 550B Hybrid Mamba-MoE Model August 28, 2026 Share Written by Aerin Kim NVIDIA's Nemotron 3 Ultra is a 550B open-weight hybrid Mamba-Attention MoE model built for long agentic runs. Here is how its architecture, training, and throughput numbers actually work. NVIDIA opened the summer of 2026 by putting its largest open-weight language model directly onto Hugging Face rather than locking it behind an API. On June 4, 2026, NVIDIA released Nemotron 3 Ultra, a 550 billion parameter hybrid Mamba-Attention Mixture-of-Experts model that sits at the top of the new Nemotron 3 family, alongside the smaller Nano and Super checkpoints [1] [2]. The headline is not just the parameter count. It is that Nemotron 3 Ultra only activates around 55 billion of those 550 billion parameters per token, roughly a 10 percent sparsity ratio, while combining state-space Mamba-2 layers with transformer attention layers in a single architecture built specifically for long-running, multi-turn agentic work [1] [3]. That combination matters because it addresses a real, specific problem that anyone building agents against a frontier model has run into: standard transformer attention gets expensive fast as a conversation, a coding session, or an agent trace grows longer, while pure state-space models like Mamba struggle to match transformer-level reasoning on dense, detail-sensitive tasks. Nemotron 3 Ultra is NVIDIA's answer to that tradeoff at genuinely frontier scale, and it ships with open weights, a permissive license, and hosted inference across seven different platforms on day one [1] [6]. This post walks through exactly how Nemotron 3 Ultra is built, why each architectural choice exists, what the independently reported benchmark numbers actually say, and how to call the model yourself through OpenRouter, NVIDIA NIM, or a self-hosted vLLM deployment. Every code block below is meant to be copied and run, not read as pseudocode. Step 1: The Nemotron 3 Family and Where Ultra Fits NVIDIA did not ship Nemotron 3 Ultra in isolation. It is the largest member of a three-tier family that NVIDIA has been rolling out since December 2025, and understanding the other two tiers is the fastest way to understand what makes Ultra different [2]. Nemotron 3 Nano shipped first, on December 15, 2025. It is a 30 billion total parameter model that activates up to 3 billion parameters per token through a hybrid latent Mixture-of-Experts design, supports a 1 million token context window, and was benchmarked by NVIDIA at roughly 4x higher token throughput than Nemotron 2 Nano while needing up to 60 percent fewer reasoning tokens to reach the same answer quality [2] [4]. Nano is the tier built for edge devices, on-device assistants, and cost-sensitive high-volume workloads where every token of latency and every dollar of inference cost compounds. Nemotron 3 Super sits in the middle: roughly 100 billion total parameters, up to 10 billion active per token, a Mixture-of-Experts design like Nano, and notably trained in 4-bit NVFP4 precision directly on NVIDIA's Blackwell architecture [2] [4]. Super is positioned as the workhorse tier, big enough for serious reasoning and agentic tasks, small enough to serve at meaningfully lower cost than Ultra. Nemotron 3 Ultra, the subject of this post, is the flagship: roughly 550 billion total parameters with up to 55 billion active per token [1] [2]. That 55-billion-active-out-of-550-billion-total ratio, about 10 percent sparsity, is deliberately close to the sparsity levels used by other very large open Mixture-of-Experts models in 2026, but the number that actually matters for a developer deciding whether to use it is not the parameter count. It is what that architecture lets the model do at inference time that a same-sized dense or standard-MoE transformer cannot, which is the subject of the next three steps. NVIDIA's own framing for why this family exists at all is worth reading directly. NVIDIA CEO Jensen Huang put it this way at launch: "With Nemotron, we're transforming advanced AI into an open platform that gives developers the transparency and efficiency they need to build agentic systems at scale" [2]. Every architectural decision covered below, the hybrid Mamba-attention backbone, the latent routing scheme, and the native speculative decoding layers, traces back to that stated goal: agentic systems that run long, multi-step traces without attention cost or routing instability making that impractical at scale. If you have been following NVIDIA's model releases, this is a genuinely different architecture from earlier Nemotron generations, and it is worth comparing against how other labs have approached scaling reasoning models. Our breakdowns of Qwen3-8-Max's 2.4 trillion parameter design and GLM-5.3's coding-focused architecture both cover competing large-scale approaches that stuck closer to a pure transformer design, which is a useful contrast to keep in mind as we get into Nemotron 3 Ultra's hybrid layers next. Step 2: Inside the Hybrid Mamba-Attention Architecture The core architectural bet in Nemotron 3 Ultra is what NVIDIA calls a Mixture-of-Experts Hybrid Mamba-Attention design: most of the network's sequence-processing layers are Mamba-2 state-space layers, interleaved with a smaller number of full transformer attention layers, all wrapped inside a Mixture-of-Experts feed-forward block [1] [3]. Why Mamba-2 Layers Exist in a 2026 Frontier Model Mamba and its successor Mamba-2 are state-space models, a different mathematical family from the attention mechanism that has defined transformers since 2017. Instead of computing a similarity score between every pair of tokens in a sequence, the way self-attention does, a state-space layer maintains a compressed running summary, a hidden state, that gets updated as each new token arrives, similar in spirit to a recurrent neural network but with a selective, input-dependent update rule that lets the model decide what to keep and what to forget as it moves through the sequence. That single design choice changes the computational cost profile completely: self-attention scales quadratically with sequence length, since every token has to compare itself against every other token, while a state-space layer scales linearly, since it only ever has to update one fixed-size hidden state per new token regardless of how long the sequence has already gotten. That difference is invisible on a short prompt and enormous on a long one. A coding agent working through a 200,000-token repository trace, or a customer support agent carrying a multi-hour conversation history, spends most of its compute reprocessing context that a pure transformer has to re-attend to at every single step. Mamba-2 layers carry that context forward in a fixed-size state instead, which is the architectural reason Nemotron 3 Ultra can serve a 262,000 token context window at full BF16 precision and push to a full 1 million tokens under NVFP4 quantization on Blackwell hardware without attention cost exploding, a point covered in depth in Step 6 below [1] [4]. Why Attention Layers Are Still There If state-space layers were strictly better, NVIDIA would not have kept any attention layers at all. They kept a smaller proportion of them because pure state-space models have a real, well-documented weakness: precise, exact retrieval of a specific fact from far back in a long context, the kind of task where a model needs to find one needle in a very large haystack rather than summarize the general shape of what came before. Full attention is exceptionally good at exactly that, because every token genuinely does get to look directly at every other token, with no compression involved. By interleaving a smaller number of true attention layers among the Mamba-2 layers, Nemotron 3 Ultra gets the linear-cost long-context efficiency of a state-space model for most of its depth, while still preserving genuine long-range, high-precision recall at the attention layers, which is exactly the profile a coding agent or research assistant needs: efficient over a long trace, but still able to pull an exact function signature or exact clause back out of a document opened fifty steps ago. Here is a simplified simulation that shows the practical effect of this hybrid layer pattern on compute cost as sequence length grows: python /code # Illustrates why interleaving a small share of full-attention layers among # linear-cost Mamba-2 layers keeps total compute far below a pure-attention # stack as sequence length grows. Attention layers scale quadratically with # sequence length; Mamba-2 layers scale linearly, the core tradeoff behind # Nemotron 3 Ultra's hybrid design [1]. def relative_cost(seq_len: int, attention_layer_ratio: float) -> float: attention_cost = attention_layer_ratio * (seq_len ** 2) mamba_cost = (1 - attention_layer_ratio) * seq_len return attention_cost + mamba_cost for seq_len in [1_000, 8_000, 64_000, 262_000]: pure_attention = relative_cost(seq_len, attention_layer_ratio=1.0) hybrid = relative_cost(seq_len, attention_layer_ratio=0.15) # ~15% attention layers savings = 1 - (hybrid / pure_attention) print(f"seq_len={seq_len:>7,} | pure-attention cost={pure_attention:>15,.0f} " f"| hybrid cost={hybrid:>15,.0f} | relative savings={savings:.1%}") Running that script at increasing sequence lengths makes the shape of the tradeoff obvious: a pure-attention stack's relative cost accelerates quadratically while the hybrid stack's cost grows close to linearly, and the gap widens fastest exactly in the range that matters for agentic workloads, tens of thousands of tokens and up. This is the same directional tradeoff that motivated hybrid attention designs elsewhere in 2026, including DeepSeek V4-Pro's Compressed Sparse and Heavily Compressed Attention split, though Nemotron 3 Ultra reaches for a genuinely different mechanism, state-space layers instead of sparsified attention, to get there. Step 3: LatentMoE Routing and Why Routing Collapse Matters A Mixture-of-Experts layer only works if tokens actually get spread across the available experts in a useful way. Give the router too little structure, and a common failure mode called routing collapse sets in, where the router learns to send most tokens to a small handful of experts it has already trained well, starving the rest of the pool of gradient signal, until the model is effectively using a much smaller fraction of its total capacity than its parameter count suggests. This is a known, well-documented failure mode in large MoE training, and it gets worse, not better, as you add more experts, which directly punishes the exact strategy, more specialists at the same active-parameter budget, that makes MoE architectures efficient in the first place. Nemotron 3 Ultra's answer is what NVIDIA calls LatentMoE routing: instead of the router looking directly at a token's raw embedding to decide which experts should handle it, it routes based on a learned latent representation of that token instead [1] [3]. A raw token embedding is a fairly rigid, high-dimensional object shaped directly by vocabulary and surface form. A latent representation, learned jointly with the rest of the network rather than fixed at the embedding layer, gives the routing decision more flexibility to group tokens by function or context rather than by superficial similarity, which in turn makes it possible to support a larger, more specialized pool of experts at the same inference cost without the router collapsing onto a small favored subset. A simplified illustration of the difference in practice: python /code import random from collections import Counter # Toy comparison of raw-embedding routing (prone to collapsing onto a small # favored subset of experts) versus latent-representation routing (spreads # load more evenly), the mechanism behind Nemotron 3 Ultra's L [truncated for AI cost control]