Skip to content
AI News HubLIVE
Original source2 min read

Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality

Summary

Zeyuan Allen-Zhu runs 64 from-scratch pretrainings (1.3–8B parameters, 1–2T Nemotron-CC tokens, roughly $100k per 8B run) to ask whether standard benchmarks capture architecturally meaningful differences. At academic scale (1.3B/100B) differences nearly vanish under careful tuning, but at 1–2T tokens Canon layers consistently improve Transformers and save about 30% of pretraining tokens at 8B for the same MMLU accuracy. New GLA5 and GDN2 variants improve gated linear attention and Gated DeltaNet with low-rank output gates; with Canon, plain gated linear attention matches Gated DeltaNet and beats Mamba2 even at 8B/1T. Yet even at that scale, linear models that match Transformers on benchmarks fail to reliably retrieve from a 500-token context, and no model beats random guessing on the simp…

SourcearXiv Computational LinguisticsAuthor: Zeyuan Allen-Zhu
Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[Submitted on 12 Nov 2026]

Title:Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality

View a PDF of the paper titled Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality, by Zeyuan Allen-Zhu

View PDF HTML (experimental)

Abstract:Do standard benchmarks see the architectural differences that matter? We study it with 64 from-scratch pretraining runs of Transformers and three linear-model families at 1.3-8B parameters on 1-2T Nemotron-CC tokens, at a cost of roughly 100k USD per 8B run.

In academic-scale pretraining (1.3B parameters, 100B tokens), architectural differences can all but disappear when the models are carefully tuned. Instead, at 1-2T tokens they become measurable: Canon layers, which add local token mixing, consistently improve Transformers, saving about 30% of pretraining tokens for the same MMLU accuracy at 8B. We also introduce GLA5 and GDN2, which improve gated linear attention and Gated DeltaNet with low-rank output gates. Surprisingly, with Canon, the simplest linear model, gated linear attention, matches Gated DeltaNet and outperforms Mamba2 even at 8B/1T. Therefore, the extra machinery of Gated DeltaNet and Mamba2 may do little more than mix neighboring tokens.

Even at 8B/1T, however, high scores on standard benchmarks can hide weaknesses in basic in-context skills. In particular, linear models that match or beat Transformers on these benchmarks cannot, in fact, reliably retrieve information from a short 500-token context. Moreover, no model, Transformer or linear, reliably beats random guessing on arguably the simplest two-hop question, stated in a few short sentences, even at this scale.

These findings $\textbf{challenge traditional architecture research}$ based primarily on standard benchmark scores. In contrast, small synthetic playgrounds, which predicted the architectural differences above at

new | recent | 2026-11

Change to browse by:

cs cs.AI

References & Citations

SciX

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • 64 from-scratch runs across Transformers and three linear-model families at 1.3–8B parameters on 1–2T Nemotron-CC tokens, at about $100k per 8B run.
  • Canon layers, which add local token mixing, consistently improve Transformers and save roughly 30% of pretraining tokens at 8B for equal MMLU accuracy.
  • GLA5 and GDN2 add low-rank output gates to gated linear attention and Gated DeltaNet; with Canon, the simplest gated linear attention matches Gated DeltaNet and outperforms Mamba2.
  • Benchmark scores can hide in-context weaknesses: linear models matching Transformers cannot reliably retrieve from a 500-token context, and no model beats random guessing on the simplest two-hop question even at 8B/1T.

Highlights and analysis are generated automatically and may contain errors. Check the original source.