Nvidia NVLink Fusion Brings Nvhbm to Next-Generation AI Infrastructure

AI factories must support increasingly large models and more complex reasoning workloads. To keep up with the insatiable compute demands of AI workloads, hyperscalers and AI-native companies are developing custom AI accelerators, or XPUs. Deploying these accelerators at scale requires high-bandwidth memory (HBM) to keep compute fed, sufficient package and silicon area for more compute, efficient power delivery, and a resilient supply chain. It also requires a rack-scale architecture for deploying XPUs into data center infrastructure. NVIDIA NVLink Fusion is the connective technology and IP that enables hyperscalers and AI natives to deploy custom XPUs and CPUs into the NVIDIA AI infrastructure platform. They can use the NVIDIA scale-up and scale-out technology stack, ecosystem, and MGX rack-scale architecture to reduce development and deployment complexity, improve performance, and accelerate time to market for semi-custom AI factories. At the package level, NVHBM, complements this unified architecture. NVHBM is a custom HBM base-die technology designed and validated with leading memory vendors that enables increased memory bandwidth, better area savings, and lower power consumption. These improvements can help custom XPUs support larger models, read KV cache data faster, and improve training and large-scale inference. Why bandwidth, die area, and power drive accelerator design Training, inference, and agentic AI workloads increasingly depend on high-throughput access to model weights, KV cache, and activation data. As AI systems scale from individual accelerators to rack-level compute domains, the accelerator package must balance compute logic, power delivery, thermal design, and high-bandwidth memory. HBM places vital memory bandwidth close to the accelerator, but qualifying leading memory technology, package integration, and validation can become a bottleneck for custom accelerator programs. Through NVLink Fusion, customers gain access to NVHBM base dies that are validated with leading memory manufacturers, helping reduce integration and qualification bottlenecks. Feature NVHBM benefit BandwidthUp to 30% more memory bandwidth compared with standard HBM4e AreaMore efficient interface connections allow up to 25% more compute die area for additional XPU capabilities PowerUp to 15% lower HBM power usage compared with standard HBM4e adds up savings across thousands of XPUs Table 1. NVHBM brings three main platform-level advantages to AI accelerator programs: higher memory bandwidth, more package and silicon area, and lower HBM power usage The memory bandwidth bottleneck in modern AI accelerators AI accelerator performance depends on how consistently compute engines are supplied with data. Higher HBM speeds increase usable memory bandwidth within a given package budget, improving the ability to serve bandwidth-intensive phases of training and inference. NVHBM delivers up to 30% more memory bandwidth per stack compared with standard HBM4e. For memory-bound or partially memory-bound AI workloads, that translates into better accelerator utilization and higher throughput. This can increase per-user token throughput during large-model inference by moving data between HBM and compute cores faster, keeping them fed. While NVHBM increases memory bandwidth within each accelerator, NVLink Fusion connects accelerators across larger domains so workloads can use distributed compute and memory more efficiently. This scale-up domain is especially critical when using advanced routing techniques like expert parallelism (EP) or WideEP. In these scenarios, different experts reside on different GPUs and require seamless, high-speed synchronization across the entire rack. NVIDIA NVLink, the scale-up networking fabric for AI factories, transfers activations and hidden states between experts and helps synchronize distributed caches across the scale-up fabric. NVHBM minimizes data starvation by keeping the local compute engines consistently fed. More package area, more flexibility For custom AI silicon, every square millimeter matters. Accelerator designers must decide how much area to allocate to matrix engines, vector units, on-chip SRAM, cache hierarchy, control logic, memory interfaces, network-on-chip, and scale-up connectivity. A custom memory implementation can help reduce the design and package overhead associated with accessing HBM, freeing up area for workload-specific capabilities. As AI workloads diversify, this additional die area gives hyperscalers more flexibility to optimize XPUs for inference serving, recommendation systems, multimodal pipelines, or internal training workloads. By reducing the area required for the memory interface, NVHBM enables teams to dedicate more of the chip directly to performance. Area savings are achieved primarily through a redesigned physical memory interface (PHY). Standard HBM relies on wider interface connections, increasing the total package footprint. NVHBM uses a custom base die optimized for efficiency, featuring reduced I/O area requirements achieved by moving the memory controller into the 3D HBM stack and integrating a custom PHY. Compared with the JEDEC HBM4e standard, this design reduces PHY and support area by up to 67%. The narrower interface also simplifies interposer routing, providing up to 80% more usable silicon across the entire layout. Figure 1. Comparison of die area savings with NVHBM compared to standard HBM