Skip to content
AI News HubLIVE
Original source2 min read

Do Value Vectors in Deep Layers Need Context from the Residual Stream?

Summary

Transformer attention layers typically produce context-dependent query, key, and value vectors from the residual stream, but this study finds that in deeper layers, learning only a context-free value vector improves model performance. The authors propose Bank of Values (BoV), which uses a token-specific lookup table for value vectors in the last third of layers, reducing compute and memory. Across 135M and 780M models, BoV improves validation loss and matches the previous best method on 21 benchmarks at 780M.

SourcearXiv Computational LinguisticsAuthor: Muyu He, Yuchen Liu, Qingya Huang, Li Zhang
Do Value Vectors in Deep Layers Need Context from the Residual Stream?
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[2606.02780] Do Value Vectors in Deep Layers Need Context from the Residual Stream?

[Submitted on 1 Jun 2026]

Title:Do Value Vectors in Deep Layers Need Context from the Residual Stream?

View a PDF of the paper titled Do Value Vectors in Deep Layers Need Context from the Residual Stream?, by Muyu He and 3 other authors

View PDF HTML (experimental)

Abstract:The success of the transformer architecture as the backbone of modern LLMs is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces context-dependent query, key, and value vectors. However, we find that model performance meaningfully improves when deeper layers learn only a context-free value vector to preserve the original token information, without drawing on any context from the residual stream. When the model has access to this context-free value vector, adding back the context-dependent component provides little additional benefit for aggregate benchmark performance. Such context-free value vectors can be stored as sparse model parameters, eliminating the need to recompute or persistently cache these values. Through systematic ablations on the key design choices for such context-free value vectors, we propose Bank of Values (BoV), a new way of computing value vectors in attention by learning a lookup table of token-specific value vectors for each of the last third of layers. Across 135M and 780M models, BoV improves validation loss over standard attention and, at 780M, the average score across 21 benchmarks, matching the previous best method that adds token information to the value vector with less compute and memory.

Comments: 13 pages, 5 figures. Code: this https URL

Subjects:

Computation and Language (cs.CL)

Cite as: arXiv:2606.02780 [cs.CL]

(or arXiv:2606.02780v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2606.02780

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Muyu He [view email] [v1] Mon, 1 Jun 2026 18:43:34 UTC (295 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Do Value Vectors in Deep Layers Need Context from the Residual Stream?, by Muyu He and 3 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-06

Change to browse by:

cs

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • Context-free value vectors in deep layers better preserve original token information and boost performance.
  • BoV learns a token-specific value vector lookup table for the last third of layers, storable as sparse parameters.
  • At 780M, BoV matches the previous best method on 21 benchmarks with less compute and memory.

Highlights and analysis are generated automatically and may contain errors. Check the original source.