2026-07-21 04:00 UTCOriginal source2 min readUpdated: 2026-07-21 08:19 UTC

It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches

A new reinforcement learning method, W2SPO, uses weak auxiliary models to inject short segments into LLM reasoning paths, improving performance and training speed on math reasoning tasks.

SourcearXiv AIAuthor: Dayu Wang, Jiaye Yang, Weikang Li, Jiahui Liang, Liwei Qian, Xin Pei, Jizhou Huang

-->

[Submitted on 7 May 2026]

Title:It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches

View a PDF of the paper titled It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches, by Dayu Wang and 6 other authors

View PDF HTML (experimental)

Abstract:Reinforcement learning with verifiable rewards has emerged as a standard approach for enhancing reasoning in large language models, which typically optimizes the policy by contrasting multiple self generated rollouts. However, we identify a critical support limited bottleneck in this paradigm: on challenging reasoning tasks, the target model's samples often exhibit semantic redundancy, converging into the same erroneous "reasoning basins" that offer negligible reward contrast for policy updates. In this paper, we propose to overcome this limitation through a weak to strong learning paradigm, where a policy's exploration is informed by a weaker but computationally efficient auxiliary model. We introduce W2SPO, an off policy RL method that injects short auxiliary segments often as brief as 8 tokens into intermediate target model trajectories and the target model then completes the reasoning path from these diverted states. Policy updates are restricted to these short inserted segments based on final verifiable rewards. Empirically, W2SPO achieves superior performance among evaluated 4B scale models on mathematical reasoning benchmarks, outperforming evaluated post trained baselines. Compared with vanilla GRPO under the same sampling budget, W2SPO improves Pass@1 from 62.3% to 64.2% while achieving a 3.55 times training speedup. These results suggest that weak auxiliary branches can induce stronger target reasoning policies by expanding local exploration support.

Subjects:

Artificial Intelligence (cs.AI)

Cite as: arXiv:2607.16205 [cs.AI]

(or arXiv:2607.16205v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2607.16205

arXiv-issued DOI via DataCite

Submission history

From: Dayu Wang [view email] [v1] Thu, 7 May 2026 13:16:09 UTC (457 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches, by Dayu Wang and 6 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.AI

new | recent | 2026-07

Change to browse by:

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)