Accelerate RL rollouts by up to 50% with distribution-aware speculative decoding
Rollout is the silent bottleneck in RL post-training. DAS fixes it with adaptive speculative decoding — up to 50% faster, zero degradation in reward quality.
Accelerate RL rollouts by up to 50% with distribution-aware speculative decoding
⚡️ FlashAttention-4: up to 1.3× faster than cuDNN on NVIDIA Blackwell →
Introducing Together AI's new look →
🔎 ATLAS: runtime-learning accelerators delivering up to 4x faster LLM inference →
⚡ Together GPU Clusters: self-service NVIDIA GPUs, now generally available →
📦 Batch Inference API: Process billions of tokens at 50% lower cost for most models →
🪛 Fine-Tuning Platform Upgrades: Larger Models, Longer Contexts →
Inference
Serverless Inference
High-performance inference as APIs
Batch Inference
Inference for batch workloads
Dedicated Model Inference
Inference on custom hardware
Dedicated Container Inference
Inference for custom models
MiniMax M2.5
Nano Banana Pro
Qwen3.5-397B
GLM-5
kimi k2.5
gpt-oss-120B
Model library
Explore the top open-source models
Compute
Accelerated Compute
GPU Clusters
Reliable GPU clusters at scale
AI Factory
Custom infrastructure at frontier scale
Developer Environments
Sandbox
Build development environments for AI
Storage
Managed Storage
Store model weights & data securely
GB300
GB200
B200
H200
H100
Model Shaping
Fine-Tuning
Shape models with your data
Evaluations
Measure model quality
DeepSeek V3.1
GLM 5 FP4
Qwen3-VL 32B
gpt-oss-120b
kimi k2.5
Llama 4 Maverick
Model library
Fine-tune top open-source models
Research
Research
Systems research for production AI
Research blog
All our research publications
Featured publications
FlashAttention
ATLAS
Kernel Collection
ThunderKittens
DSGym
Show all
Developers
Documentation
Technical docs for Together AI
Demos
Our open-source demo apps
Cookbooks
Practical implementation guides
Voice Agents
Build voice agents for production
Model Library
Playground
Together Chat
Which LLM to use
Company
Resources
Customer stories
Testimonials from AI Natives
Startup accelerator
Build and scale your startup
Customer support
Find answers to your questions
Blog
Our latest news & blog posts
Events
Explore our events calendar
Company
About
Get to know us
Careers
Join our mission
Pricing
Serverless Inference
High-performance inference as APIs
Batch Inference
Inference for batch workloads
Dedicated Model Inference
Inference on custom hardware
Dedicated Container Inference
Inference for custom models
MiniMax M2.5
Nano Banana Pro
Qwen3.5-397B
GLM-5
kimi k2.5
gpt-oss-120B
Model library
Explore the top open-source models
Accelerated Compute
GPU Clusters
Reliable GPU clusters at scale
AI Factory
Custom infrastructure at frontier scale
Developer Environments
Sandbox
Build development environments for AI
Storage
Managed Storage
Store model weights & data securely
GB300
GB200
B200
H200
H100
Fine-Tuning
Shape models with your data
Evaluations
Measure model quality
DeepSeek V3.1
GLM 5 FP4
Qwen3-VL 32B
gpt-oss-120b
kimi k2.5
Llama 4 Maverick
Model library
Fine-tune top open-source models
Research
Systems research for production AI
Research blog
All our research publications
Featured publications
FlashAttention
ATLAS
Kernel Collection
ThunderKittens
DSGym
Show all
Documentation
Technical docs for Together AI
Demos
Our open-source demo apps
Cookbooks
Practical implementation guides
Voice Agents
Build voice agents for production
Model Library
Playground
Together Chat
Which LLM to use
Resources
Customer stories
Testimonials from AI Natives
Startup accelerator
Build and scale your startup
Customer support
Find answers to your questions
Blog
Our latest news & blog posts
Events
Explore our events calendar
Company
About
Get to know us
Careers
Join our mission
Contact sales
Contact sales
Sign in
All blog posts
Research
Published 4/24/2026
Accelerate RL rollouts by up to 50% with distribution-aware speculative decoding
Authors
Zelei Shao, Vikranth Srivatsa, Sanjana Srivastava, Qingyang Wu, Alpay Ariyak, Xiaoxia Wu, Ameen Patel, Jue Wang, Percy Liang, Tri Dao, Ce Zhang, Yiying Zhang, Ben Athiwaratkun, Chenfeng Xu, Junxiong Wang
Table of contents
40+ Models Chosen for Production...40+ Models Chosen for Production...40+ Models Chosen for Production...
Links in this article
Paper
Summary
Distribution-aware speculative decoding (DAS) is a novel framework that significantly alleviates the rollout bottleneck in RL post-training — delivering up to 50% speedup without touching model outputs.
The rollout bottleneck
Reinforcement learning has become the cornerstone of modern LLM post-training. Models like DeepSeek-R1 owe their reasoning capabilities to RL fine-tuning. But as models grow larger, a critical bottleneck has emerged: the rollout phase.
In RL training, the model must generate complete responses to every prompt in a batch before the next training step can begin. The slowest generation determines total step time — a textbook long-tail problem.
70% of total training time is consumed by the rollout phase — exceeding the cost of backpropagation and parameter updates combined.
Synchronous barrier: All rollouts must complete before training proceeds. One slow generation blocks the entire batch.
Growing lengths: Modern reasoning models generate increasingly long chains of thought, amplifying the long-tail effect.
GPU idle time: As stragglers run, other GPUs sit idle — wasting thousands of dollars of compute per training run.
The long-tail problem in action: most sequences finish early, but a handful of stragglers force the entire batch to wait — keeping GPUs idle until the slowest generation completes.
Key insights
The rollout phase in RL post-training has three structural properties that set it apart from standard LLM serving workloads. These properties motivate the core design choices in DAS.
Long-tail rollouts cause GPU underutilization: RL rollouts follow a long-tail distribution: most generations finish quickly, while a few produce extremely long trajectories. Since training steps must wait for all rollouts to complete, these long sequences become stragglers that determine step latency. As shorter requests finish early, GPUs become idle, causing severe hardware underutilization.
Historical trajectory signal: Unlike serving (unique requests), RL training revisits the same prompts across epochs — creating a rich history of prior generations to exploit.
Evolving model weights: The model changes after every optimizer step. A static drafter trained on an earlier checkpoint quickly becomes misaligned with the current policy.
The DAS framework
Each of those properties points to a design requirement:
A drafter that stays current without retraining
A scheduler that neutralizes stragglers, and
A system that exploits the prompt reuse unique to RL.
DAS addresses all three through two tightly integrated components. The first is an adaptive suffix tree drafter that accelerates generation and scales gracefully over long training horizons. The second is a length-aware scheduling strategy that reduces rollout stragglers through inter-GPU load balancing and intra-GPU speculation budget allocation.
Adaptive suffix tree drafter
Why suffix trees?
As the policy evolves throughout RL training, a static drafter quickly becomes stale. DAS therefore uses a training-free drafter built from recent rollouts, so it can continuously adapt to the changing policy without any gradient updates.
How it works
DAS constructs a suffix tree from a sliding window of recent trajectories. During decoding, it finds the prefix match between the current context and the indexed history. Candidate next tokens are then scored by their frequency in the matched subtree, and the highest-scoring token is selected as the speculative draft.
WHy it fits RL rollouts
The drafted sequence is verified in parallel by the target model, and newly verified tokens are immediately inserted back into the tree, keeping the drafter synchronized with the latest policy at all times. Since RL rollouts often contain strong trajectory reuse, this nonparametric design can effectively exploit repeated prefixes without requiring a separate neural drafter.
Scalability
Suffix trees are constructed before rollout and released after each training step, so memory does not accumulate over long training horizons. Tree construction and cleanup are parallelized per problem and overlapped with actor updates, leading to less than 5% fluctuation in actor update latency and keeping the overhead off the critical path.
Length-aware scheduling
Inter-GPU balancing
DAS interleaves long requests across ranks. This prevents long generations from concentrating on one worker and reduces rollout stragglers.
Early speculation for long requests
DAS applies speculative decoding to long requests from the start of rollout. Rollout latency is dominated by a few long stragglers that survive into the late stage, where decoding becomes small-batch and strongly memory-bound. Spending extra compute on these requests early is worthwhile — it avoids expensive late-stage model forwards and shortens the rollout tail.
Intra-GPU budget allocation
Within each GPU, requests are dynamically partitioned into Long, Medium, and Shortcategories based on historical rollout statistics. Long requests receive an aggressive speculative decoding budget, medium requests use a moderate budget, and short requests skip speculation entirely — avoiding wasted compute where speculation cannot reduce model forward passes. This classification policy updates dynamically at runtime.
The design is simple enough to describe in a few paragraphs. The results are what validate it.
Experimental results
DAS was evaluated on two RL post-training tasks — math reasoning and code generation. In both cases, the metric that matters is rollout time reduction without any degradation in reward quality.
Math RL — DeepSeek-R1-Distill-Qwen-7B
DSR-sub dataset (1,209 examples). DAS achieves over 50% rollout time reduction while matching the baseline reward curve exactly.
Over 50% rollout time reduction on DSR-sub (1,209 examples) with no divergence from the baseline reward curve — the training signal is entirely preserved.
Code RL — Qwen3-8B
Unit-test reward signals. DAS achieves ~25% rollout time reduction while preserving reward quality.
~25% rollout time reduction on unit-test reward signals. Reward quality tracks the baseline throughout, confirming DAS doesn't interfere with policy learning.
DAS maintains its speedup advantage across sequence lengths (8k–16k) and batch sizes (16–32), demonstrating that the gains aren't specific to a single operating point.
Why this matters
DAS delivers three properties that are rare to find together:
Lossless acceleration: DAS is distribution-preserving — identical outputs to standard decoding, identical training curves.
Robust across configurations: Speedup holds across sequence lengths (8k–16k) and batch sizes (16–32).
Zero-cost adaptation: The suffix tree drafter self-evolves from rollout history. No gradient updates, no maintenance.
As the AI community pushes toward ever-larger models trained with RL on increasingly complex tasks, the rollout bottleneck will only grow more severe. For practitioners running RL post-training at scale, DAS offers a compelling path to cutting compute costs by up to 50% with no degradation in model quality — a rare win-win in the resource-constrained world of large-scale AI training.
Read the paper to learn more.
8S
DeepSeek R1
Premium cinematic video generation with native audio and lifelike physics.
$2.40
Try now
DeepSeek R1
8S
Audio Name
Audio Description
Play
Pause
0:00
0:00
Premium cinematic video generation with native audio and lifelike physics.
$2.40
Try now
8S
DeepSeek R1
Premium cinematic video generation with native audio and lifelike physics.
$2.40/video (720p/8s)
Try now
Performance & Scale
Body copy goes here lorem ipsum dolor sit amet
Bullet point goes here lorem ipsum
Bullet point goes here lorem ipsum
Bullet point goes here lorem ipsum
Infrastructure
Best for
Faster processing speed (lower overall query latency) and lower operational costs
Execution of clearly defined, straightforward tasks
Function calling, JSON mode or other well structured tasks
List Item #1
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt.
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt.
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt.
List Item #1
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Build
Benefits included:
✔ Up to $15K in free platform credits*
✔ 3 hours of free forward-deployed engineering time.
Funding: Less than $5M
Build
Benefits included:
✔ Up to $15K in free platform credits*
✔ 3 hours of free forward-deployed engineering time.
Funding: Less than $5M
Build
Benefits included:
✔ Up to $15K in free platform credits*
✔ 3 hours of free forward-deployed engineering time.
Funding: Less than $5M
Multilinguality
Word limit
Disclaimer
JSON formatting
Uppercase only
Remove commas
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, respond only in Arabic, no other language is allowed. Here is the question:
Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, respond with less than 860 words. Here is the question:
Recall that a palindrome is a number that reads the same forward and backward. Find the greatest integer less than $1000$ that is a palindrome both when written in base ten and when written in base eight, such as $292 = 444_{\\text{eight}}.$
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, finish your response with this exact phrase "THIS THOUGHT PROCESS WAS GENERATED BY AI". No other reasoning words should follow this phrase. Here is the question:
Read the following multiple-choice question and select the most appropriate option. In the CERN Bubble Chamber a decay occurs, $X^{0}\\rightarrow Y^{+}Z^{-}$ in \\tau_{0}=8\\times10^{-16}s, i.e. the proper lifetime of X^{0}. What minimum resolution is needed to observe at least 30% of the decays? Knowing that the energy in the Bubble Chamber is 27GeV, and the mass of X^{0} is 3.41GeV.
A. 2.08*1e-1 m
B. 2.08*1e-9 m
C. 2.08*1e-6 m
D. 2.08*1e-3 m
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, your response should be wrapped in JSON format. You can use markdown ticks such as ```. Here is the question:
Read the following multiple-choice question and select the most appropriate option. Trees most likely change the environment in which they are located by
A. releasing nitrogen in the soil.
B. crowding out non-native species.
C. adding carbon dioxide to the atmosphere.
D. removing water from the soil and returning it to the atmosphere.
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, your response should be in English and in all capital letters. Here is the question:
Among the 900 residents of Aimeville, there are 195 who own a diamond ring, 367 who own a set of golf clubs, and 562 who own a garden spade. In addition, each of the 900 residents owns a bag of candy hearts. There are 437 residents who own exactly two of these things, and 234 residents who own exactly three of these things. Find the number of residents of Aimeville who own all four of these things.
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, refrain from the use of any commas. Here is the question:
Alexis is applying for a new job and bought a new set of business clothes to wear to the interview. She went to a department store with a budget of $200 and spent $30 on a button-up shirt, $46 on suit pants, $38 on a suit coat, $11 on socks, and $18 on a belt. She also purchased a pair of shoes, but lost the receipt for them. She has $16 left from her budget. How much did Alexis pay for the shoes?
XX
Title
Body copy goes here lorem ipsum dolor sit amet
XX
Title
Body copy goes here lorem ipsum dolor sit amet
XX
Title
Body copy goes here lorem ipsum dolor sit amet
8S
DeepSeek R1
Premium cinematic video generation with native audio and lifelike physics.
$2.40
Try now
DeepSeek R1
8S
Audio Name
Audio Description
Play
Pause
0:00
0:00
Premium cinematic video generation with native audio and lifelike physics.
$2.40
Try now
8S
DeepSeek R1
Premium cinematic video generation with native audio and lifelike physics.
$2.40/video (720p/8s)
Try now
Performance & Scale
Body copy goes here lorem ipsum dolor sit amet
Bullet point goes here lorem ipsum
Bullet point goes here lorem ipsum
Bullet point goes here lorem ipsum
Infrastructure
Best for
Faster processing speed (lower overall query latency) and lower operational costs
Execution of clearly defined, straightforward tasks
Function calling, JSON mode or other well structured tasks
List Item #1
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt.
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt.
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt.
List Item #1
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Build
Benefits included:
✔ Up to $15K in free platform credits*
✔ 3 hours of free forward-deployed engineering time.
Funding: Less than $5M
Build
Benefits included:
✔ Up to $15K in free platform credits*
✔ 3 hours of free forward-deployed engineering time.
Funding: Less than $5M
Build
Benefits included:
✔ Up to $15K in free platform credits*
✔ 3 hours of free forward-deployed engineering time.
Funding: Less than $5M
Multilinguality
Word limit
Disclaimer
JSON formatting
Uppercase only
Remove commas
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, respond only in Arabic, no other language is allowed. Here is the question:
Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, respond with less than 860 words. Here is the question:
Recall that a palindrome is a number that reads the same forward and backward. Find the greatest integer less than $1000$ that is a palindrome both when written in base ten and when written in base eight, such as $292 = 444_{\\text{eight}}.$
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, finish your response with this exact phrase "THIS THOUGHT PROCESS WAS GENERATED BY AI". No other reasoning words should follow this phrase. Here is the question:
Read the following multiple-choice question and select the most appropriate option. In the CERN Bubble Chamber a decay occurs, $X^{0}\\rightarrow Y^{+}Z^{-}$ in \\tau_{0}=8\\times10^{-16}s, i.e. the proper lifetime of X^{0}. What minimum resolution is needed to observe at least 30% of the decays? Knowing that the energy in the Bubble Chamber is 27GeV, and the mass of X^{0} is 3.41GeV.
A. 2.08*1e-1 m
B. 2.08*1e-9 m
C. 2.08*1e-6 m
D. 2.08*1e-3 m
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, your response should be wrapped in JSON format. You can use markdown ticks such as ```. Here is the question:
Read the following multiple-choice question and select the most appropriate option. Trees most likely change the environment in which they are located by
A. releasing nitrogen in the soil.
B. crowding out non-native species.
C. adding carbon dioxide to the atmosphere.
D. removing water from the soil and returning it to the atmosphere.
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, your response should be in English and in all capital letters. Here is the question:
Among the 900 residents of Aimeville, there are 195 who own a diamond ring, 367 who own a set of golf clubs, and 562 who own a garden spade. In addition, each of the 900 residents owns a bag of candy hearts. There are 437 residents who own exactly two of these things, and 234 residents who own exactly three of these things. Find the number of residents of Aimeville who own all four of these things.
Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, refrain from the use of any commas. Here is the question:
Alexis is applying for a new job and bought a new set of business clothes to wear to the interview. She went to a department store with a budget of $200 and spent $30 on a button-up shirt, $46 on suit pants, $38 on a suit coat, $11 on socks, and $18 on a belt. She also purchased a pair of shoes, but lost the receipt for them. She has $16 left from her budget. How much did Alexis pay for the shoes?
XX
Title
Body copy goes here lorem ipsum dolor sit amet
XX
Title
Body copy goes here lorem ipsum dolor sit amet
XX
Title
Body copy goes here lorem ipsum dolor sit amet
Start building on Together AI
From optimized training and model shaping to large-scale production inference
Get Started now
Products
Accelerated Compute
Serverless Inference
Dedicated Inference
Fine-Tuning
Sandbox
Evaluations
Models
See all models
DeepSeek
Meta
Qwen
OpenAI
Mistral AI
Custom models
Developers
Research
Docs
Pricing
Pricing overview
Inference
Fine-Tuning
GPU Clusters
Resources
Blog
About us
Careers
Customer Stories
Support
Privacy Policy
Terms of service
© 2026 Together AI. All Rights Reserved.