2026-07-22 04:00 UTCOriginal source2 min readUpdated: 2026-07-22 08:27 UTC

Convolution for Large Language Models

This paper studies whether lightweight depthwise convolutions can provide local inductive bias to LLMs without materially increasing model size. Macro-level ablation on Qwen3 Transformer blocks finds optimal placement of convolution on projected queries, keys, and values before attention. Micro-level study favors a residual depthwise convolution with kernel size k=3 without extra normalization or activation. Across Qwen3 models and data budgets, this design improves average accuracy on seven downstream benchmarks while adding less than 0.01% parameters. A case study suggests convolution makes repeated token IDs more sensitive to immediate context.

SourcearXiv Computational LinguisticsAuthor: Yuchuan Tian, Yingte Shu, Wei He, Shuo Zhang, Tianchen Zhao, Chao Xu, Xinghao Chen, Yunhe Wang, Hanting Chen, Yu Wang

-->

[Submitted on 20 Jul 2026]

Title:Convolution for Large Language Models

View a PDF of the paper titled Convolution for Large Language Models, by Yuchuan Tian and 9 other authors

View PDF HTML (experimental)

Abstract:Large language models (LLMs) largely rely on Transformers, where self-attention provides global token interaction but does not explicitly encode the locality of natural language. We study whether lightweight depthwise convolutions can supply this local inductive bias without materially increasing model size. Our macro-level ablation compares convolution at 17 locations in a Qwen3 Transformer block and finds the best results when convolution is applied to the projected queries, keys, and values before attention. A subsequent micro-level study favors a residual depthwise convolution with kernel size $k=3$, without additional normalization or activation. Across Qwen3 models and several pre-training data budgets, this design improves the average accuracy on seven downstream benchmarks while adding less than $0.01\%$ parameters. A representation-level case study further suggests that the convolution makes repeated token IDs more sensitive to their immediate context. These results support depthwise convolution as a lightweight complement to self-attention for modeling short-range token interactions.

Comments: 12 pages, 5 figures

Subjects:

Computation and Language (cs.CL)

Cite as: arXiv:2607.18413 [cs.CL]

(or arXiv:2607.18413v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2607.18413

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yuchuan Tian [view email] [v1] Mon, 20 Jul 2026 18:02:25 UTC (314 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Convolution for Large Language Models, by Yuchuan Tian and 9 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-07

Change to browse by:

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)