2026-05-29原文

The Cognitive Categorical Transformer: Category-Theoretic Inductive Biases for Language Modeling

The Cognitive Categorical Transformer (CCT) is a 306M-parameter architecture that augments GPT-2 Small with cognitive and category-theoretic components, achieving 21.27 perplexity on WikiText-103, a 2.92 (12%) reduction over a fine-tuned baseline. Ablations attribute 84% of the improvement to GT-Full simplicial message passing. The study also identifies a structure/consistency distinction among categorical priors.

Article intelligence

EngineersAdvanced

Key points

CCT achieves 21.27 perplexity on WikiText-103, 2.92 lower than GPT-2 Small baseline.
Ablation studies attribute 84% of the gain to GT-Full simplicial message passing.
Consistency-style categorical priors show no benefit, supporting a structure/consistency distinction.

Why it matters

This matters because CCT achieves 21.27 perplexity on WikiText-103, 2.92 lower than GPT-2 Small baseline.

Technical impact

May affect model selection, inference cost, product capability, and evaluation benchmarks.

[2605.28864] The Cognitive Categorical Transformer: Category-Theoretic Inductive Biases for Language Modeling

[Submitted on 22 May 2026]

Title:The Cognitive Categorical Transformer: Category-Theoretic Inductive Biases for Language Modeling

View a PDF of the paper titled The Cognitive Categorical Transformer: Category-Theoretic Inductive Biases for Language Modeling, by Al Kari

View PDF HTML (experimental)

Abstract:The Cognitive Categorical Transformer (CCT) is a 306M-parameter architecture that augments a pretrained GPT-2 Small backbone with cognitively grounded components derived from category theory and several inspirations from cognitive science. Under a matched-step protocol (215,000 optimizer steps, matched data, matched optimizer and schedule) on WikiText-103, CCT reaches 21.27 validation perplexity, compared with 24.19 for an identically fine-tuned GPT-2 Small baseline. The architecture therefore contributes a 2.92 PPL (12% relative) reduction beyond what in-domain fine-tuning alone provides. A retrain-from-scratch ablation that holds GT-Full simplicial message passing bypassed across the entire seven-phase activation schedule reaches 23.72 PPL, localizing 84% of the architectural improvement (2.45 of 2.92 PPL) to GT-Full. We present the first ablation-validated evidence that simplicial message passing improves language-model perplexity at the 306M-parameter scale on WikiText-103. Published GPT-2 Large reaches 22.05 zero-shot PPL on WikiText-103 with 6.2x more parameters than GPT-2 Small; this paper treats that number as an external published reference, not as the architectural benchmark. Three negative results on consistency-style categorical priors (sheaf smoothing, adjunction round-trip, curvature regularization) and the joint structural-prior result for GT-Full and PrecisionWeightedPP together support an empirical pattern termed the *structure/consistency distinction*, in which categorical priors that add new topology improve language modeling and those that enforce a consistency identity do not.

Subjects:

Artificial Intelligence (cs.AI); Computation and Language (cs.CL)

Cite as: arXiv:2605.28864 [cs.AI]

(or arXiv:2605.28864v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2605.28864

arXiv-issued DOI via DataCite

Submission history

From: Al Kari [view email] [v1] Fri, 22 May 2026 00:56:20 UTC (106 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled The Cognitive Categorical Transformer: Category-Theoretic Inductive Biases for Language Modeling, by Al Kari

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.AI

new | recent | 2026-05

Change to browse by:

cs cs.CL

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)