待翻譯：Thoughts about Scaling Law (from Z.ai Cofounder)

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

jietang (@jietang): "Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions. The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed. Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter. Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it. This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count. Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more." | XCancel @jietang 3h Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions. The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed. Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter. Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it. This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count. Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more. Aug 19, 2026 · 5:04 AM UTC 79 224 1,653 115,757 Sort replies: Relevant Recent Liked @CamusMyth 18m Replying to @jietang 壓縮即智能，壓縮極限我認為應該是在幾B甚至更少，在這個過程中我們拓展參數不過是因為還沒找到最好的壓縮方法。後訓練很重要，在長程任務中摸清模型邊界，通過後訓練去拓展邊界，再重新去在更難的任務中去找邊界，形成loop 256 @Tape_Vector 35m Replying to @jietang $NVDA benefits when better post training and inference efficiency expand AI demand beyond parameter races 77 @luyun0120 2h Replying to @jietang 值得深思。這個問題背後有更深的結構，表面的答案往往只是冰山一角。能問出好問題的人，比能給出答案的人更稀缺。 4 1,365 @FanShifu 3h Replying to @jietang 唐老師，post-training 什麼時候會有自己的 Chinchilla？預訓練當年有 Kaplan 走偏後被糾正。也可能這顆旋鈕的餘量永遠用不完，畢竟環境可以一直生成。很想知道 GLM 內部看到的 RL 曲線長什麼樣，是已經平了，還是仍有餘力 3 1,204 @GeekParkHQ 1h Replying to @jietang curious how much of “we turned this dial because it had the most slack” is measured beforehand, and how much is clear only in hindsight? 101 @idovmamane 43m Replying to @jietang im honestly uneasy how obsessed everyone is getting with parameter counts, like as if more always means better. feels like we're missing the nuance around actual data quality and real task performance. but i still find it wild how scaling laws are actually revealing deeper patterns, not just for AI but potentially how we learn across different fields 169 @Synapse_Brief 2h Replying to @jietang The terminal-bench jump from 4.6 to 28.3 is the headline, but the real efficiency signal is the 22% drop in output tokens at max effort — 96K down to 75K — while pushing task completion from 23.4% to 34.5%. That is not just better reasoning. That is a post-training pipeline that learned to compress its own chain-of-thought without sacrificing accuracy. The 744B total / 40B active MoE split is identical to GLM-5.2. Every gain came from scaling RL environments and the slime async framework, not from touching the base weights. Z.ai is effectively running a controlled experiment on the Roberts et al. finding that reasoning favors more data while memorization favors parameters. They kept parameters fixed and flooded the post-training stage with long-horizon tasks. The cyber capability emerging faster than the training schedule predicted is the piece that should worry infrastructure people. A model that reasons across multi-stage exploit chains is not a better retrieval engine. It is a better causal chain carrier. That is exactly the capability the original post identifies as not living in total parameter count. GLM-5.3 is the production proof. 462 @OrganeticSphere 40m Replying to @jietang One implication is that the control surface has to scale along different axes too. If the base model and architecture stay fixed, yet a month of long-horizon post-training materially changes agent behavior, parameter count tells us very little about what downstream systems should actually trust. Capability scaling is becoming multidimensional. The evidence and validation boundaries around those capabilities need to evolve with it. 134 @defaultsettle 3h Replying to @jietang The operational lesson is just as important: Same base. Same architecture. Same total and activated parameters. Different post-training. Materially different behavior. Parameter count is not model identity. For production agents, each evaluation and receipt must bind the exact checkpoint, post-training version, inference configuration, tools and task conditions. An agent approved on GLM-5.2 should not silently inherit the same authority when moved to GLM-5.3 simply because the architecture is unchanged. Scaling has multiple dials. Changing any dial that changes behavior creates a new verification boundary. 627 @hanqing1120 53m Replying to @jietang 參數競賽讓位給旋鈕選擇。同底座只轉後訓練就出收益，這個控制實驗比堆萬億參數更有説服力。 92 1h Replying to @jietang Next time, maybe mid-training, pre-training, and even more. 唐老師説了，控制變量實驗做完了，下次，都有。 方法論無敵！ 820 @ragzoi 2h Replying to @jietang parameter count is the model-card equivalent of counting lines of code. it says something, just not whether the thing survives a long-running agent task. 431 1h Replying to @jietang Amazing work and insights, thanks for open sourcing and for sharing your wisdom. 121 3h Replying to @jietang 👏 1 1,319 @buswe_com 3h Replying to @jietang Justo, la arquitectura y la calidad de los datos mueven la aguja tanto como el recuento de parámetros, y salen mucho más baratas de escalar. 1 1,919 @ShinkaIoT 1h Replying to @jietang The over-training-for-inference point is the underrated half of scaling talk, most labs still optimize like a model gets evaluated once instead of called a billion times a day. 27 3h Replying to @jietang 非常好的補充角度 1 1,277 @saurabh_here1 1h Replying to @jietang Where do algorithmic advancements fit into this? I have seen models converge much faster/with less data, using better loss formulations, RL, etc. Can we define a Pareto fronti [truncated for AI cost control]