翻訳待ち：With software alone, one B200 beats the LPU and gets close to Cerebras

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

Every year now somebody ships new silicon built for inference. Cerebras put the weights on a wafer, Groq built the LPU, SambaNova built the RDU, and all three of them are real machines that move tokens fast. I like that this is happening. Inference is worth designing hardware for. I spent the last two years studying NVIDIA and AMD GPU architecture to build a code editor for GPU kernels. Reading those manuals all day teaches you what a card can do, and then you look at what people are getting out of the same card and the two numbers do not match. So I stopped trying to write every kernel by hand and started building agents that write them. I open sourced AutoKernel, which generates and tunes GPU kernels, small and readable so you can actually follow what it does. Then AutoMegaKernel, which fuses a whole decode step into persistent kernels so the intermediates never leave the chip. All of it is for the same thing, getting models to run properly on the cards people already own. That is why the leaderboards bother me. When I see a GPU serving 423 tokens per second on a model whose byte budget I can work out on paper, my first thought is not that the GPU is slow. My first thought is that nobody wrote the software. So I decided to check whether I could close that gap on a single B200 without touching anything but the software. Cerebras runs gpt-oss-120b at 1,991 tokens per second. The fastest GPU provider you can buy from is Google Vertex at 423, then Databricks at 324 and Azure at 300. Cerebras calls that a 5x lead over GPUs and they are right, it is 4.7x (cerebras.ai/blog/blackwell-vs-cerebras). Cerebras, in their own words (cerebras.ai/blog/blackwell-vs-cerebras). The 1,991 is a hardware answer and a good one. They keep the weights in SRAM instead of HBM, so the bottleneck the rest of us fight does not exist on their machine. I am not going to argue with that and I did not beat it. The 423 is the number I went after. I rented a single B200 on Modal, which is what I reach for when I want a GPU without thinking about it, and started measuring. Out of the box I got 411 tokens per second. I changed four settings and the same card running the same weights gave me 1,366, and single prompts hit 2,215. I did not write a kernel and I did not recompile anything. Stock SGLang 411 tok/s 26 percent of this card's memory bandwidth After four settings 1,366 tok/s 72 percent, and 3.2x the fastest GPU provider Best single prompt 2,215 tok/s above Cerebras, on one rentable card That is 3.2x past the fastest GPU provider on the board. It clears Groq at 476 and SambaNova at 708, and it puts one rented card at 69 percent of the wafer. The abstraction lies to you, and it is not lying about the hardware Every layer between you and the chip is there to stop you thinking about the chip, and it works. You call generate, tokens come out at some rate, and that rate feels like the speed of the machine. It is not. It is the speed of whatever defaults you happened to call. That is why people blame the hardware when something is slow, and why they are almost always wrong. The framework cannot tell you that it is the thing holding you back. It throws no error, everything looks fine, and the number it gives you is steady and repeatable, so it feels like physics when it is a config file. The only way out is to work out what the card can do yourself, from bytes and bandwidth, and compare. That number does not care what framework you use. For gpt-oss-120b on a B200 the arithmetic is short. Decoding one token reads every active weight exactly once, which is 1.91 GB of attention in bf16, 1.90 GB of top-4 routed experts in mxfp4, and 1.16 GB of lm_head in bf16, so 4.97 GB per token. A B200 has 8 TB/s of HBM3e (nvidia.com/en-us/data-center/dgx-b200), so the floor is 0.621 milliseconds per token and the ceiling is about 1,610 tokens per second. Stock SGLang took 2.43 milliseconds per token. Seventy four percent of every token was not reading weights. Red is the irreducible read of 4.97 GB of weights at 8 TB/s, which caps this model at 1,610 tok/s. Grey is everything else. The chip sat idle three quarters of the time. Every instinct that says buy the faster card is aimed at the 26 percent that was already working. What I measured, and how I ran everything on a single B200 on Modal with 178.4 GiB of HBM3e and 148 SMs. Model is openai/gpt-oss-120b, which ships natively in MXFP4 and activates about 5.1B of its 120B parameters per token. Serving through SGLang 0.5.16, greedy at temperature zero, single stream, one prompt at a time, four fixed prompts, three repetitions, median. The number reported is decode rate excluding prefill, which is how Artificial Analysis defines Output Speed, the average number of tokens received per second after the first token is received (artificialanalysis.ai/methodology). I picked this model because all three ASIC vendors publish on it, so the comparison is the same weights and the same metric. Artificial Analysis, gpt-oss-120b, read 6 August 2026 (artificialanalysis.ai/models/gpt-oss-120b/providers) Providertok/sSilicon Cerebras1,991wafer SambaNova708RDU Groq476LPU Google Vertex423GPU Databricks324GPU Azure300GPU Amazon101GPU CoreWeave33GPU The full board. Everything from Google Vertex down is running GPUs, and the spread inside that GPU group is 13x on identical model weights. Look at the GPU half of that chart on its own. Thirteen companies run the same open weights on cards any of them can rent and they land 13x apart, from 423 down to 33. They all have the same silicon, so every bit of that 13x is settings, kernels and serving code. Software is already worth 13x between people who bought the same hardware, before anyone sits down and tunes anything on purpose. The ladder Median of three repetitions over four prompts, same card Steptok/s stock defaults411.2 latency knobs429.9 n-gram speculative decoding628.5 speculative_attention_mode=decode697.1 wide n-gram search1,165.1 and 1,366.2 The grey bars are the settings anyone can copy. The red one is where the n-gram search width lands it. Nothing recompiled and no kernel written. The first rung is small. SGLang defaults stream_interval to 1, so it detokenizes and dispatches on every single token, and scheduler_recv_interval to 1, so it polls every iteration. On a 0.621 millisecond budget that is real money, and fixing it bought 4.5 percent. The second rung is speculative decodingA draft proposes several tokens and the target model verifies them in one forward pass, so every accepted token is produced without a separate full read of the weights., the same trick that gave Groq its six times. A draft proposes several tokens, the target verifies them in one forward pass, and every accepted token is a token produced without a separate 4.97 GB weight read. I could not use a trained draft head, for reasons in the failures section, so this is n-gram speculationDrafting by matching patterns in the text generated so far, rather than with a trained draft model. It needs no extra weights, and it pays off exactly as much as the output repeats itself., which drafts by matching patterns in the text generated so far. The third rung was a flag I had never touched. speculative_attention_mode defaults to prefill, and setting it to decode gave 9.7 percent. The last row is two numbers because I ran that configuration twice and got both, a 17 percent spread on identical settings. N-gram acceptance depends on what the trie has built up, so the same config lands in a different place each run. If I quoted the better one I would be picking a number, not measuring one. The rung that mattered was found by breaking it I was told, reasonably, that the CPU n-gram proposer was serializing with the GPU and that its search cost was the bottleneck. So I cut it down, max_bfs_breadth from 10 to 2 and max_trie_depth from 18 to 8, expecting overhead to fall. Throughput collapsed to 433.0. So I ran the knob the other way, and the curve has a sharp peak. speculative_ngram_max_bfs_breadth, single stream max_bfs_breadthtok/s 2433.0 10, the default697.1 241,165.1 321,110.4 48341.1 Wider search costs CPU and buys accepted tokens, and each accepted token skips a 4.97 GB read. Past 24 the trade stops paying. The shipped default is 10, and you can read it straight off the documentation. The n-gram parameters from the SGLang speculative decoding docs (docs.sglang.io/docs/advanced_features/speculative_decoding). Maximum BFS breadth, default 10. For single-stream decode that leaves about 40 percent on the floor, and past 24 the CPU search stops paying for itself and falls off a cliff. The default is not a mistake. It is set for throughput serving, where many requests share the GPU and CPU time is tight. It is the wrong default for one user waiting on one stream, and nothing in the stack tells you which of the two you are doing. Where that lands One rented B200 against the custom silicon Providertok/sRatio one B200, two runs1,165.1 to 1,366.2 Groq4762.45x to 2.87x faster SambaNova7081.65x to 1.93x faster Cerebras1,9910.59x to 0.69x Per prompt, the two runs measured 447.8, 1930.8, 1882.5, 219.1 and then 525.3, 2207.0, 2215.6, 291.9 tokens per second. Four of those eight readings are above Cerebras's published 1,991, on one rentable GPU. Acceptance tracks predictability. Repeatable output flies and novel prose does not, under one fixed configuration. That spread is the most useful thing in the post, so I am showing it instead of hiding behind a median. N-gram speculation drafts by matching text it has already produced, so structured reasoning where phrasing repeats runs above 2,200 while open-ended prose that never repeats itself runs at 219. The trick pays off exactly as much as the output is predictable. A vendor reporting one number is averaging over their own prompt mix, and you cannot see this shape at all. What speculation does and does not change Greedy verification is what keeps this honest. A drafted token is accepted only where it matches the token the target model would have produced on its own, and on the first mismatch the rest of the draft is thrown away and the target's token is taken instead. So speculation under greedy verification is exact, not approximate. It changes how many forward passes you spend, not which tokens come out. What did not work, which is most of it Fifteen configurations, and the failures tell you more than the wins. Both published EAGLE-3 draft heads scored below running with no speculation at all, 417.6 for NVIDIA's and 325.0 for the SGLang team's, because acceptance was near zero. NVIDIA trained theirs against their NVFP4 checkpoint and this is the MXFP4 one, and a draft that never guesses right costs you the draft pass for nothing. Getting NVIDIA's head to load at all is worth recording. It failed with a tensor mismatch, 8640 against 5760, which is three times the hidden size against two. SGLang v0.5.16 maps the draft's requested capture layers with an off-by-one, so the last of three requested layers falls outside the loop, and the same version silently drops the checkpoint's input_norm weight because it never builds that module. Both were fixed upstream in commit 5df193b4ac, merged six days after v0.5.16 was tagged. A nightly with the fix loaded it, and it still lost. Every attention backend except triton is unavailable here. FlashAttention 3 requires SM 80 to 90 and a B200 is SM100. FA4 forces a page size of 128 and trtllm_mha forces 64, and SGLang then refuses to run a speculative tree wider than one token on a paged backend because it produces incorrect results. I was glad to hit that guard instead of quietly getting wrong answers. The triton MoE runner OOMs at 156 GiB because it dequantizes MXFP4 back to bf16. torch.compile asserts and then OOMs on the same path. DFLASH with a real draft model reached 493.5, better than nothing an [truncated for AI cost control]