AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。
Oct 06 2026 MoE Math Demystified: What Does 8x7B Actually Mean? Daria SobolevaEtienne Goffinet MoE Fundamentals | Router Wars | Debugging Dead MoE Models | MoE at Scale | MoE Math Demystified This video breaks down MoE inference arithmetic and deployment bottlenecks across different hardware setups. If you can’t open the video displayed above, please use this link to open it on YouTube: https://youtu.be/gHpDBoyCOrE What does 8x7B actually mean? You probably thought it meant 8 experts with 7B active parameters per token. We did too. Turns out it is actually 13B active parameters. But wait, where does 13B come from? This is exactly the kind of confusion this post clears up (skip to the answer). We'll explain what those numbers actually mean for inference by answering how much memory you need, how many GPUs, and what the commonly hit bottlenecks are in production deployment. We'll show that single-GPU deployment is memory-bound, multi-GPU setups are communication-bound, and specialized hardware like Cerebras WSE is compute-bound. Originally, we set out to write a simple post on MoE arithmetic. Then we kept digging. And digging. What started as basic math turned into a full explanation of inference bottlenecks, hardware architectures, and deployment strategies. Welcome to MoE inference 101. The title stayed, but the scope didn't. Up to this point, the Mixture-of-Experts (MoE) series has focused on training aspects. In part 3, you trained your own MoE model and scaled it in part 4 to production size. Now what? We shift to inference. During inference, model weights are frozen, no gradients or optimizer states. This sounds much simpler compared to training mode. But! Despite doing less work, MoE inference has its own challenges. Fun fact: people who train models rarely think about inference costs, and vice versa. Your authors are no exception, but we are trying to be better. So, if you're deploying MoE models, or you trained an MoE model and want to know what your design choices have led to, or you simply want to understand how to run MoE inference efficiently, keep scrolling. How much memory do I need? Want to avoid hitting an OOM error during inference deployment? Let’s examine two components that dominate memory space: model weights and kv-cache. To simplify our calculations, we are going to use standard modern transformer setup. We use RoPE positional embeddings (Su et al., 2023), SwiGLU nonlinearity (Shazeer, 2020; Dauphin et al., 2016), layer norms, multi-head attention, untied embeddings, and industry standard learned routing (Soboleva, 2025a). Model weights Let's walk through the math. First, we'll calculate a single decoder block memory requirement, then account for all layers , and finally add the remaining network parameters not included in the decoder blocks (bottom to top). Bias terms are omitted from the following equations as they are negligible (regardless of the model size). Our MoE model consists of an embedding layer, followed by decoder blocks, and an unembedding layer. Each decoder block contains two layer norms, an attention layer, a router, and an MoE layer with expert networks (Figure 1). Figure 1: Visual breakdown of MoE model decoder architecture. The MoE model consists of an embedding layer, followed by decoder blocks, and an unembedding layer. Each decoder block includes two layer norms, an attention layer, a router, and an MoE layer with expert networks. We start with embedding weights: The embedding layer consists of an input embedding matrix of size . Following the embedding layer, next comes the layer norm: You can see that we have a multiplier here. This is because we have two layer norms per decoder block, and each layer norm stores gains and biases of the same size . Next, let's do the math for the attention layer: In attention, we account for four weight matrices (query, key, value, and output) each with size . Perfect, now comes the first MoE component, the router: In the case of learned routing, we need to store a weight matrix of size with learnable router weights. The last thing we need to account for is the MoE layer weights: Each expert network uses SwiGLU nonlinearity and thus requires three linear transformations (gated linear unit, up-projection, and down projection). With dynamic routing, we can’t predict which experts will be activated, so you must provision enough device memory to store all experts. Thus, MoE models require more memory capacity than dense networks with equivalent active parameters. Finally, we use untied embeddings, so we need to account for an additional memory of an unembedding matrix of size : Combining all of the weights together in the decoder block, we get: Accounting for decoder blocks in the network, and adding embedding and unembedding weights, we get the total amount of memory we need in bytes to hold model weights: Next, let's answer the question of how much compute we need to run inference. How much compute do I need? Just as we estimated the amount of memory required to run inference with our MoE model, here we will focus on estimating the number of FLOPs (Floating Point Operations). Later, we will convert our FLOPs into throughput and latency metrics. These metrics will help us answer how fast our MoE model will run. There are two important stages in inference. The first one is called prefill. During prefill, the model processes the entire prompt all at once. During the second stage, we start generating our answer. With autoregressive transformers, generation happens one token at a time. Each new token depends on the tokens previously generated. We call the second stage decode. Let's focus on calculating the FLOPs we spend during the prefill stage and then do a separate analysis for decode. Most of the operations in the transformer consist of matrix-vector, matrix-matrix, and Hadamard computations. So let's establish how many FLOPs are needed for these base operations. Prefill Cool, now we are ready to dive into the actual computations. Note that we will ignore an embedding layer. It is usually implemented as lookup table and doesn't require arithmetic operations. We start with layer norms: We need 7 FLOPs per element (for sequence with tokens and hidden dimension ) to calculate layer norm FLOPs. This comes from computing the mean and variance across the embedding dimension, then performing normalization and scaling. Feel free to do the math on your own and check us! The multiplier is accounting for both layer norms. Let's move on to the attention FLOPs: Here we need to account for multiplications between the input matrix of size and each of the query, key, and value projection matrices (each of size ). After attention computation, we multiply the attention output with the output projection matrix (also of size ). These four matrix multiplications give us the first term in equation . The second term calculates FLOPs for the attention computation itself. We calculate attention logits (by multiplying query and key projection vectors of size each), requiring FLOPs. Then we multiply the result with the value projection vector. Thus, the total number of FLOPs for these two matrix-matrix computations equals FLOPs (second term). The final term accounts for softmax operations, which require approximately FLOPs per element (exponentiation, normalization, and scaling). Since we perform separate softmax operations for each of the attention heads over elements, this contributes (third term). Next, let's account for RoPE positional embeddings: With RoPE, we add an additional matrix-matrix multiplication in the attention layer between the query projection vector of size and the rotation matrix of size . Due to the sparsity in the rotation matrix, this dot product can be decomposed into the sum of two Hadamard products between matrices of size (Su et al., 2021), which gives us FLOPs. Following Black et al. (2022), we perform rotations only on of the embedding vector dimensions, which leads us to total FLOPs. Next come the FLOPs we spend in the router network: With industry standard learned routing, we need to multiply an input matrix of size by the matrix of size , giving us the first term in the equation (12). The second term accounts for the softmax FLOPs that we spend for each token across the sequence with elements, across all experts . The next layer to account for in our FLOPs calculations is MoE layer: In the MoE layer, each expert contains two consecutive feed-forward layers (up and down projections). The up projection with SwiGLU performs two matrix-matrix operations, multiplying the input matrix of size by the weight matrix of size . This requires FLOPs. The down projection performs one matrix-matrix operation, multiplying the intermediate result of size by a weight matrix of size , requiring FLOPs. Now, we were able to calculate the first term in the equation 13. For SwiGLU nonlinearity, we also need to perform a Hadamard product between two matrices of size , requiring FLOPs. Swish function requires 5 FLOPs per activation (negation, exponentiation, addition, division, and multiplication), thus we need remaining FLOPs (second term). Note that we only activate out of experts per token, thus MoE models require significantly less fewer FLOPs than dense networks with the same total parameter count. The reduced compute means faster training and inference. The last layer we need to account for is the unmbedding layer: It transforms our token hidden representation back into the vocabulary space (multiplying an input vector of size by the matrix of size ). Overall, during the prefill stage, we require the total number of FLOPs to spend per sequence: Decode During decode phase we're operating token-by-token level, not on full sequences. This means that all the computations from the prefill stage get reduced by a factor of . Except attention. Here we have a problem. In attention, we need to maintain the context of all previous tokens in our window. So even though we are processing a single token, we still need to compute pairwise relationships between all tokens in the sequence that came before it. Pretty soon it blows up (we're talking growth rate as the context length increases). Check equation (10) because this is exactly how many FLOPs you will spend on just a single token. This is clearly unsustainable. To avoid this, we'll trade memory for compute and introduce a kv-cache. With kv-cache we'll cache and vectors from all the previous tokens in the sequence, so we never have to recompute them. Now, attention compute will look like this: Notice what happened. The quadratic term is gone. We cached all the past token-token relationships needed for attention. We only compute the new token relationship to all those present in the sequence. Now compute grows linearly with . Much better. Thus, overall FLOPs we spend during decode per token: For MoE models, kv-cache memory requirements are identical to those of dense networks. Following the analysis from (Chen, 2022), the additional memory that we need to provision can be estimated with: We are accounting for 2 vectors ( and ) that we store across layers , across the full sequence , with separate vectors for each of the attention heads, where each vector has a dimension . Why my GPU doesn’t go brrr? We've calculated how much memory we need in section 2, and how many FLOPs our model requires in section 3. Our theory looks solid, but if you've run experiments, you know the math doesn't always check out in practice. Super frustrating. So let's verify everything together and figure out why GPUs often sit idle instead of going brrr. We'll load Qwen3-30B-A3B (an MoE with 3B active and 30B total parameters) (Yang et al., 2025) onto an H100 GPU [1] and measure how fast it generates tokens at the decode stage: Figure 2: Measuring how fast Qwen3-30B-A3B generates tokens at the decode stage on H100 G [truncated for AI cost control]