2026-05-29原文

From Context Shift to Stylistic Collapse: Why Training Objectives Matter More Than Scale

A new paper analyzes 17 LLMs (410M-100B+ parameters) and documents that instruction-tuned systems systematically collapse language entropy along discourse and structural dimensions (mean amplification: 1,949-16,853%, peaks: 5,181-209,675%), while suppressing complex punctuation to 3.2-23.2% of baseline. These effects do not worsen under RLHF. Weak intervention (lambda=1.0) exacerbates collapse by 240%, while strong control (lambda=5.0) achieves 40.5% improvement and outperforms frontier models by 96.7-98.2% despite 200-1000x scale disadvantage. Strong control also delivers 15% higher distinct-4, 27% higher vocabulary diversity, and 78% lower repetition than moderate regularization. The findings underscore that alignment requires sufficient control strength, not merely distributional smoothing.

Article intelligence

EngineersAdvanced

Key points

Instruction tuning causes language entropy collapse along discourse and structural dimensions, with significant suppression of complex punctuation.
RLHF does not worsen stylistic collapse, but weak regularization exacerbates it.
Strong control (lambda=5.0) outperforms frontier models despite large scale disadvantage.
Alignment requires sufficient control strength, not just distributional smoothing.

Why it matters

This matters because instruction tuning causes language entropy collapse along discourse and structural dimensions, with significant suppression of complex punctuation.

Technical impact

May affect model selection, inference cost, product capability, and evaluation benchmarks.

[2605.28826] From Context Shift to Stylistic Collapse: Why Training Objectives Matter More Than Scale

[Submitted on 8 Apr 2026]

Title:From Context Shift to Stylistic Collapse: Why Training Objectives Matter More Than Scale

View a PDF of the paper titled From Context Shift to Stylistic Collapse: Why Training Objectives Matter More Than Scale, by Rohan Mahapatra

View PDF HTML (experimental)

Abstract:In modern LLMs, linguistic features function not as stylistic artifacts but as probes of probability mass, allocated under training alignment objectives. Language models trained with contemporary pipelines exhibit severe reshaping of linguistic features, leading to extreme language re-distribution. While previous stylometric analyses explored linguistic differences between AI-generated and human texts, we focus on the reshaping plaguing the LLM training pipeline itself. We analyze 17 models (410M-100B+ parameters) across 24 linguistically-motivated probes, documenting that instruction-tuned systems systematically collapse language entropy along discourse and structural dimensions (mean amplification: 1,949-16,853%, peaks: 5,181-209,675%), while selectively suppressing complex punctuation to 3.2-23.2% of baseline frequencies. These effects do not worsen under RLHF, as divergence patterns are statistically indistinguishable (p > 0.25) across matched base and instruction-tuned model pairs. Weak intervention (lambda=1.0) exacerbates collapse by 240%, while strong control (lambda=5.0) achieves 40.5% improvement and outperforms frontier models by 96.7-98.2% despite 200-1000x scale disadvantage. Additionally, lambda=5.0 delivers 15% higher distinct-4, 27% higher vocabulary diversity, and 78% lower repetition than moderate regularization, establishing that alignment requires sufficient control strength, not merely distributional smoothing. Our findings underscore how modern LLMs reallocate stylistic probability mass, despite RLHF and scale. More broadly, our work reveals a structural limitation of current alignment pipelines: preference optimization reshapes language distributions invisible to standard quality metrics yet detectable through distributional probes, with implications for AI detection, training data contamination, and long-term linguistic evolution.

Comments: 26 pages, 13 tables, 2 figures. Planning to submit to NeurIPS 2026

Subjects:

Computation and Language (cs.CL)

Cite as: arXiv:2605.28826 [cs.CL]

(or arXiv:2605.28826v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2605.28826

arXiv-issued DOI via DataCite

Submission history

From: Rohan Mahapatra [view email] [v1] Wed, 8 Apr 2026 02:13:46 UTC (770 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled From Context Shift to Stylistic Collapse: Why Training Objectives Matter More Than Scale, by Rohan Mahapatra

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-05

Change to browse by:

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)