Microsoft's Maia 200 AI Accelerator at Hot Chips 2026

Facebook X Pinterest Linkedin ReddIt Email Print Copy URL Microsoft-Maia200-Hero The fourth AI accelerator presentation of Hot Chips 2026 comes from Microsoft, who like so many other hyperscalers has gone into the business of designing their own AI accelerator. Now in their second generation, Microsoft is preparing to deploy their Maia 200 accelerator inside their Azure data centers. The 3nm chip was announced early this year, while today’s presentation will be the deepest Microsoft has gone into the chip’s architecture and the many design tweaks they have made to improve its performance over the original Maia 100. This article is being written live from the presentation, so please excuse any typos. Microsoft’s Maia 200 Accelerator at Hot Chips 2026 Maia is notable right off the bat for not being quite as aggressive of a design in terms of chip size. Though by no means a small chip at 140 billion transistors, Microsoft has paired the accelerator with just 6 stacks of HBM3e memory, and the total chip TDP is just 750 Watts. Though as these are designed to go into racks (just as with any other AI accelerator), the performance and power consumption still add up quickly. Microsoft Maia 200 Hot Chips 2026 Overview Today is going to the story of how Microsoft designed the Maia 200. The talk will go over the goals of the architecture, the SoC and its architecture, and how it comes together to form the larger system, with software layered in on top. Microsoft Maia 200 Hot Chips 2026 Inference Design Goals Inference has a very diverse workload between prefill and decode. Mixture of expert modeling also creates a diversity of workloads. These and more were the design target of the Maia 200. Microsoft Maia 200 Hot Chips 2026 SDLA Dataflow Architecture Maia 200 uses the Software Defined Local Acces (SLDA) dataflow architecture. Microsoft makes a distinction of architecture along two dimensions: data access and data movement. SDLA is defined by the fact that the dataflow is software-defined, while data access is completely local. This organization is similar to the Flynn’s classic taxonomy of processors. The SDLA architecture affords a high degree of determinism. The dataflow is controlled in software, and thus is set at compile time which means it is known is advance. Meanwhile Microsoft is able to avoid a lot of cross-talk and heavy bandwidth/fabric needs by keeping data access within the compute elements. Microsoft Maia 200 Hot Chips 2026 Async Control & Data Path Here is an abstract look at the control and data path flow of the architecture. DMA and processing elements run the data path; each one has its own hardware queue of instructions. Explicit sync is achieved via semaphores. Control, I/O, and compute are all running independently, and being chained/synced via the aforementioned semaphores. Microsoft Maia 200 Hot Chips 2026 High Throughput Compute Microsoft’s fundamental computing block is a tile. Each tile contains a tile tensor unit (TTU) their matrix math engine, the tile vector processor (TVP) with SIMD engines, and then the tile control processor (TCP). Rounding out th etile is a TDMA block, and then an L1 cache block. Microsoft Maia 200 Hot Chips 2026 Hierarchial Data Storage Microsoft employs a hierarchical memory structure, with L1 (tiler) and L2 (cluster) cache layers below the memory layer. Data is only moved between layers when it is needed to do so. Microsoft Maia 200 Hot Chips 2026 IO As for I/O, Microsoft uses a custom NIC and interconnect protocol as their network fabric. Reliability is a first-class design parameter here, with hardware-based recovery mechanisms. This custom design is achieved with minimal die space on Maia 200. Each SoC has 8 Ethernet lanes, split up into four network planes. Microsoft Maia 200 Hot Chips 2026 Load Balancing Maia 200 also heavily emphasizes load balancing both inside a chip and across all of the chips in a cluster. Effective load balancing is fundamental to allowing Maia 200 to scale up to large domains. Microsoft Maia 200 Hot Chips 2026 Maia 200 All Together Maia 200 is built on TSMC 3nm, with a 750 Watt TDP. There is 7TB/second of HBM bandwidth, accomplished via 6 HBM stacks. In terms of compute throughput, the chip offers 10,000 TFLOPS of FP4 performance. The SoC die is 820mm2 altogether. Microsoft Maia 200 Hot Chips 2026 System FCQ Topology Microsoft uses a fully connected quad topology for their networking. Microsoft Maia 200 Hot Chips 2026 Unified Ethernet Scale-Up There is no scale-out networking for Maia 200. It is all scale-up networking. Microsoft’s slide demonstrates 128 racks with 6000 chips all together. Microsoft took a bet on unified Ethernet here, and it has paid off with operational simplicity. Each rack has tier-0 switches to connect the nodes within the racks, and then tier-1 switches to connect the racks themselves. Microsoft Maia 200 Hot Chips 2026 IO Stack Shifting over to the software side of Microsoft’s ecosystem, they didn’t just co-design the chip and system, but they co-designed the entire stack right on through the software. The software at the core of this is the Microsoft Collective Communication Library (MCCL). Microsoft Maia 200 Hot Chips 2026 Kernel Co-Design The SDLA design of the hardware requires that the kernels running on the hardware be tuned for the hardware for the best performance. Microsoft Maia 200 Hot Chips 2026 Kernel Batch GEMM GEMM workloads are one of the fundamental workloads run on Maia 200. Above, Microsoft’s presentation lays out how a batch GEMM workload is handled on the chip. A gather operation takes place over the central GNOC, while the weights are loaded from HBM, and finally the output is pinned to the L1 SRAM. Microsoft Maia 200 Hot Chips 2026 Kernel Distributed GEMM Of course, Maia 200 chips aren’t meant to be used alone; they are meant to be used together in a scale-up domain. So here is a look at the distributed workflow for batch GEMMs. Microsoft Maia 200 Hot Chips 2026 Kernel Design Attention Data reuse is also a very important element of execution on Maia 200 for efficiency purposes, as it reduces the memory traffic flowing across the trip. Block interleaving is used to get fully overlapped tensor/SIMD operations. Microsoft Maia 200 Hot Chips 2026 Kernel Design Collectives Microsoft uses an adaptive logical topology for collectives. A collective can complete with a single hop and no overhead. Microsoft Maia 200 Hot Chips 2026 Adaptive Topology Maia also supports a ring-based topology for when large transfers are needed. Microsoft Maia 200 Hot Chips 2026 GEMM Performance Now here is a look at the performance of the Maia 200 in GEMM benchmarks. Points on the left are memory-bound, points on the right are compute-bound. Maia gets very close to its theoretical maximum (roofline) in these benchmarks. Microsoft Maia 200 Hot Chips 2026 Attention Performance Meanwhile attention benchmarks show an effective peak of 1.65 PFLOPS. Microsoft Maia 200 Hot Chips 2026 Collective Performance Finally, here is a look at collective benchmarking. Microsoft is able to reach almost 1.3TB/second in BF16 AllReduce (with the chart also showing how the different broadcast implementations affect performance), while All2All peaks at 655 GB/second. Microsoft Maia 200 Hot Chips 2026 Summary And that is a quick look at Maia 200. Microsoft has opted for a data-movement-centric architecture with SDLA, and paired this with a unified all-Ethernet networking structure to keep the networking simple. Software is the third pillar of Maia 200, with the classical software development house co-designing the hardware and software in concert to optimize the very explicit flow of data.