2026-07-09 04:00 UTCOriginal source2 min readUpdated: 2026-07-09 08:15 UTC

Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention

This paper investigates how the spectral properties of the attention score matrix are influenced by positional encoding. Analyzing seven pretrained models, they find that previous-token heads under RoPE exhibit rotational spectra, while those under learned-absolute and ALiBi do not. Dynamic analysis shows spectral signatures emerge after behavior, and causal experiments demonstrate that no spectral channel is necessary but bans delay formation.

SourcearXiv Machine LearningAuthor: Li Hengyu (Institute for Solid State Physics, The University of Tokyo)

Article intelligence

EngineersAdvanced

Key points

RoPE positional encoding yields rotational spectra for attention heads, while absolute and ALiBi yield non-rotational spectra.
Spectral signatures emerge after the attention behavior, not before.
Causal experiments show no spectral channel is necessary; models can route around bans but incur formation delays.
Positional scheme sets the default spectral algebra of the attention solution, acting as a fingerprint after function rather than a hard constraint.

Why it matters

This matters because roPE positional encoding yields rotational spectra for attention heads, while absolute and ALiBi yield non-rotational spectra.

Technical impact

May affect model selection, inference cost, product capability, and evaluation benchmarks.

This panel is AI-generated and reviewed for accuracy.

-->

[Submitted on 7 Jul 2026]

Title:Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention

View a PDF of the paper titled Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention, by Li Hengyu (Institute for Solid State Physics and 1 other authors

View PDF HTML (experimental)

Abstract:The pre-softmax score of an attention head is a bilinear form $score(i,j) = x_i^T M x_j$ in a learned operator $M = W_q^T W_k$. Because M is generally non-symmetric, hence non-normal, it has a complex eigenspectrum and non-orthogonal eigenvectors, the regime where non-Hermitian and random-matrix tools apply. We ask what this spectrum encodes, at three levels for previous-token and induction circuits. Statically, across seven pretrained models spanning three positional schemes, the strongest previous-token heads are spectrally rotational under RoPE and non-rotational, or content-like, where position enters outside QK (learned-absolute and ALiBi); the model-level separation is perfect at every top-k examined (exact permutation $p=0.029$), and zeroing the per-frequency RoPE phase $Im(M_t)$ eliminates induction on a pre-identified previous-token head in all three RoPE models. Dynamically, over public Pythia checkpoints every head originates at the random-matrix (Ginibre) null; the rotational signature emerges with the behavior, not before it, and the population-median suppression that yields the final profile follows circuit formation, so the profile is a consolidated fingerprint, not a precursor. Causally, and at toy scale, no spectral channel is necessary: constrained two-layer training reroutes around every ban with capability intact, albeit at a significant formation delay (four pre-registered contrasts, $q_BH

new | recent | 2026-07

Change to browse by:

cond-mat cond-mat.dis-nn cs

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)