待翻译：Thoughts about Scaling Law (from Z.ai Cofounder)

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

jietang (@jietang): "Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions. The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed. Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter. Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it. This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count. Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more." | XCancel @jietang 3h Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions. The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed. Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter. Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it. This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count. Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more. Aug 19, 2026 · 5:04 AM UTC 79 224 1,653 115,757 Sort replies: Relevant Recent Liked @CamusMyth 18m Replying to @jietang 压缩即智能，压缩极限我认为应该是在几B甚至更少，在这个过程中我们拓展参数不过是因为还没找到最好的压缩方法。后训练很重要，在长程任务中摸清模型边界，通过后训练去拓展边界，再重新去在更难的任务中去找边界，形成loop 256 @Tape_Vector 35m Replying to @jietang $NVDA benefits when better post training and inference efficiency expand AI demand beyond parameter races 77 @luyun0120 2h Replying to @jietang 值得深思。这个问题背后有更深的结构，表面的答案往往只是冰山一角。能问出好问题的人，比能给出答案的人更稀缺。 4 1,365 @FanShifu 3h Replying to @jietang 唐老师，post-training 什么时候会有自己的 Chinchilla？预训练当年有 Kaplan 走偏后被纠正。也可能这颗旋钮的余量永远用不完，毕竟环境可以一直生成。很想知道 GLM 内部看到的 RL 曲线长什么样，是已经平了，还是仍有余力 3 1,204 @GeekParkHQ 1h Replying to @jietang curious how much of “we turned this dial because it had the most slack” is measured beforehand, and how much is clear only in hindsight? 101 @idovmamane 43m Replying to @jietang im honestly uneasy how obsessed everyone is getting with parameter counts, like as if more always means better. feels like we're missing the nuance around actual data quality and real task performance. but i still find it wild how scaling laws are actually revealing deeper patterns, not just for AI but potentially how we learn across different fields 169 @Synapse_Brief 2h Replying to @jietang The terminal-bench jump from 4.6 to 28.3 is the headline, but the real efficiency signal is the 22% drop in output tokens at max effort — 96K down to 75K — while pushing task completion from 23.4% to 34.5%. That is not just better reasoning. That is a post-training pipeline that learned to compress its own chain-of-thought without sacrificing accuracy. The 744B total / 40B active MoE split is identical to GLM-5.2. Every gain came from scaling RL environments and the slime async framework, not from touching the base weights. Z.ai is effectively running a controlled experiment on the Roberts et al. finding that reasoning favors more data while memorization favors parameters. They kept parameters fixed and flooded the post-training stage with long-horizon tasks. The cyber capability emerging faster than the training schedule predicted is the piece that should worry infrastructure people. A model that reasons across multi-stage exploit chains is not a better retrieval engine. It is a better causal chain carrier. That is exactly the capability the original post identifies as not living in total parameter count. GLM-5.3 is the production proof. 462 @OrganeticSphere 40m Replying to @jietang One implication is that the control surface has to scale along different axes too. If the base model and architecture stay fixed, yet a month of long-horizon post-training materially changes agent behavior, parameter count tells us very little about what downstream systems should actually trust. Capability scaling is becoming multidimensional. The evidence and validation boundaries around those capabilities need to evolve with it. 134 @defaultsettle 3h Replying to @jietang The operational lesson is just as important: Same base. Same architecture. Same total and activated parameters. Different post-training. Materially different behavior. Parameter count is not model identity. For production agents, each evaluation and receipt must bind the exact checkpoint, post-training version, inference configuration, tools and task conditions. An agent approved on GLM-5.2 should not silently inherit the same authority when moved to GLM-5.3 simply because the architecture is unchanged. Scaling has multiple dials. Changing any dial that changes behavior creates a new verification boundary. 627 @hanqing1120 53m Replying to @jietang 参数竞赛让位给旋钮选择。同底座只转后训练就出收益，这个控制实验比堆万亿参数更有说服力。 92 1h Replying to @jietang Next time, maybe mid-training, pre-training, and even more. 唐老师说了，控制变量实验做完了，下次，都有。 方法论无敌！ 820 @ragzoi 2h Replying to @jietang parameter count is the model-card equivalent of counting lines of code. it says something, just not whether the thing survives a long-running agent task. 431 1h Replying to @jietang Amazing work and insights, thanks for open sourcing and for sharing your wisdom. 121 3h Replying to @jietang 👏 1 1,319 @buswe_com 3h Replying to @jietang Justo, la arquitectura y la calidad de los datos mueven la aguja tanto como el recuento de parámetros, y salen mucho más baratas de escalar. 1 1,919 @ShinkaIoT 1h Replying to @jietang The over-training-for-inference point is the underrated half of scaling talk, most labs still optimize like a model gets evaluated once instead of called a billion times a day. 27 3h Replying to @jietang 非常好的补充角度 1 1,277 @saurabh_here1 1h Replying to @jietang Where do algorithmic advancements fit into this? I have seen models converge much faster/with less data, using better loss formulations, RL, etc. Can we define a Pareto fronti [truncated for AI cost control]