2026-07-01 04:00 UTCOriginal source2 min readUpdated: 2026-07-01 08:01 UTC

Hierarchical Global Attention (HGA)

Hierarchical Global Attention (HGA) is a drop-in replacement for dense causal attention in pretrained long-context transformers, enabling 64K token context on a single RTX 5090 without retraining or calibration, with minimal quality loss.

SourcearXiv Machine LearningAuthor: Woernle Frank, Fedosov Vladimir, Grinenko Artemiy

-->

[Submitted on 29 Jun 2026]

Title:Hierarchical Global Attention (HGA)

View a PDF of the paper titled Hierarchical Global Attention (HGA), by Woernle Frank and 2 other authors

View PDF HTML (experimental)

Abstract:Hierarchical Global Attention (HGA) is a drop-in replacement for dense causal attention in pretrained long-context transformers. HGA preserves the original checkpoint parameters: the pretrained $W_Q$, $W_K$, $W_V$, and $W_O$ projections remain unchanged, no calibration parameters are introduced, and no retraining is required.

Applied to Qwen3-30B-A3B-Instruct-2507-FP8 on a single RTX~5090 (32GB), the patched model runs out of the box at a 64K-token context, where token-level K/V storage is not feasible on this hardware.

Unlike previous sparse-attention methods, HGA performs hierarchical two-level routing. It first retrieves relevant chunks using compact RoPE-aware summaries and then refines the selection by routing only the most relevant groups before performing exact token-level attention. This hierarchical retrieval significantly reduces the number of fetched tokens while preserving exact attention over the retrieved token set, making RAM- and NVMe-backed storage practical.

The full historical token K/V resides in host RAM or NVMe storage, while only a small routed working set is transferred to GPU memory during attention. Consequently, GPU memory consumption depends primarily on model weights and the routed working set rather than on the total context length.

Across all tested context lengths (4K - 64K tokens), routed attention remains within approximately $0.01$--$0.02$ nats of dense attention while the sparsity used is just about 3%. These results suggest that the approximation introduced by hierarchical routing is small, and that the remaining quality gap is likely dominated by long-context positional encoding rather than by the routing algorithm itself.

Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

MSC classes: Primary: 68T45, Secondary: 68T07

Cite as: arXiv:2606.30709 [cs.LG]

(or arXiv:2606.30709v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2606.30709

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Vladimir Fedosov [view email] [v1] Mon, 29 Jun 2026 16:20:35 UTC (13 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Hierarchical Global Attention (HGA), by Woernle Frank and 2 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.LG

new | recent | 2026-06

Change to browse by:

cs cs.AI

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)