2026-07-10 04:00 UTCOriginal source2 min readUpdated: 2026-07-10 08:17 UTC

Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

Jet-Long introduces a tuning-free zero-shot method for extending LLM context windows by using dynamic bifocal RoPE, which adapts the rescaling factor to sequence length, achieving high efficiency and strong performance on multiple benchmarks.

SourcearXiv Machine LearningAuthor: Haozhan Tang, Zerui Wang, Yuxian Gu, Song Han, Han Cai

Article intelligence

EngineersAdvanced

Key points

Existing zero-shot context extension methods use a fixed rescaling factor, leading to trade-offs between short and long contexts.
Jet-Long employs dynamic bifocal RoPE with local and long-range windows, automatically adjusting the rescaling factor based on sequence length.
The method is inference-efficient, with minimal overhead, and achieves state-of-the-art results on RULER, HELMET-RAG, and PG-19.
Jet-Long generalizes to hybrid attention architectures without retraining and is robust to hyperparameter choices.

Why it matters

This matters because existing zero-shot context extension methods use a fixed rescaling factor, leading to trade-offs between short and long contexts.

Technical impact

May affect model selection, inference cost, product capability, and evaluation benchmarks.

This panel is AI-generated and reviewed for accuracy.

-->

[Submitted on 8 Jul 2026]

Title:Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

View a PDF of the paper titled Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE, by Haozhan Tang and 4 other authors

View PDF HTML (experimental)

Abstract:Modern LLMs are increasingly deployed in long-context applications such as retrieval-augmented generation, repository-level coding, and agentic workflows whose accumulated reasoning and tool traces routinely push the input an order of magnitude past the pretraining window, making zero-shot context extension the dominant deployment path for open-weight checkpoints. Most existing zero-shot methods fix a single rescaling factor up front, so an aggressive factor sacrifices short-context fidelity while a conservative one breaks down at long contexts. We propose Jet-Long, a tuning-free zero-shot method that pairs a local RoPE-faithful window with a long-range window whose rescaling factor adapts dynamically to the current sequence length, recovering the base model exactly at short inputs while extrapolating cleanly at long ones. An inclusion-exclusion attention merge and an on-the-fly RoPE correction rotation make the bifocal construction essentially free at inference; fused into a single CuTe kernel, long-context prefill reaches up to $1.39\times$ FA2 throughput on H100 (approaching the Hopper-only FA4), and single-batch generation incurs $\le 4\%$ overhead at every length. On Qwen3-1.7B/4B/8B up to 128K context, Jet-Long leads RULER by $+4.79$/$+2.18$/$+2.03$~pp over the strongest baseline at 1.7B/4B/8B, achieves the best overall accuracy on HELMET-RAG (a benchmark identified by HELMET as the most efficient predictor of downstream long-context performance) and attains the lowest PG-19 perplexity. Jet-Long also generalizes to hybrid attention architectures such as Jet-Nemotron for further long-context improvement without retraining, and remains hyperparameter-resilient for ease of deployment.

Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

Cite as: arXiv:2607.07740 [cs.LG]

(or arXiv:2607.07740v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2607.07740

arXiv-issued DOI via DataCite

Submission history

From: Haozhan Tang [view email] [v1] Wed, 8 Jul 2026 06:23:42 UTC (699 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE, by Haozhan Tang and 4 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.LG

new | recent | 2026-07

Change to browse by:

cs cs.AI

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)