In large-model reinforcement learning (RL), samplers usually run on the same accelerators and often the same precision as the trainer, keeping the two numerically close for stable training. Outside of RL, however, serving systems are not constrained in this manner: they’re free to optimize generation for speed and cost using low-precision formats such as FP8, MXFP4 and NVFP4, co-designed kernels, and even accelerators built just for long-context prefill.
Since sampling already takes 80–85% of the GPUs in large RL runs, this leaves a lot on the table! We ask whether we can use mixed-precision, mixed-accelerator samplers, each running its own kernels, while keeping training stable and performant.
Samplers on six kinds of accelerators, each with its own precision and kernel, feed one BF16 trainer on B200.
Two problems stand in the way.
The first problem is learning. Each of these optimizations adds to training–inference mismatch, making samples more off-policy, which often destabilizes training. Off-policy correction is the natural solution; it lets a trainer learn from data generated by a different policy, such as stale samples in asynchronous training. Since it makes no assumption about why the two policies differ, it should apply equally to different precisions or accelerators.
The second problem is the system. RL infrastructure is usually built around clusters of homogeneous accelerators. Mixed samplers require the opposite: autoscaling accelerators that burst when needed and run using their own precisions and kernels.
Through off-policy correction and our autoscaling, serverless GPUs, we show that stable, performant, and cheaper RL is possible with mixed-precision, mixed-accelerator samplers by training an open-weight model fully asynchronously on agentic coding tasks.
Our setup
We train Qwen3.6-35B-A3B on agentic coding with up to 256K tokens of context using the 2.7K repository tasks Xiaomi released with MiMo-V2.6, after removing the repository history and other traces through which some tasks leak the reference fix. In each episode, the model creates a patch for an issue over multiple turns and submits it through a bash tool in a Modal Sandbox. It receives a reward of 1 if the submitted patch passes the task’s tests in a fresh sandbox, and 0 otherwise. We evaluate on SWE-bench Pro V2, which has no overlap with the training set, and its hard subset HARD-51 separately. Every checkpoint is evaluated in BF16.
We use Miles as the RL framework, with SGLang samplers and a Megatron-LM trainer that updates the policy in BF16 on B200 GPUs. Training is fully asynchronous with no limit on staleness, and each update uses 128 prompts with 8 trajectories each. The samplers keep their own accelerators, precisions and kernels, and the trainer does not replay their MoE routing decisions.
The samplers run in one of two setups:
The mixed samplers span six kinds of accelerators: B200 and B300 serve NVFP4 weights, H100 and H200 serve FP8 and BF16 weights, and A100 and RTX PRO 6000 serve BF16 weights. Each sampler runs its accelerator’s own kernels and keeps its KV cache in FP8. A router assigns each agent session to one of these eight groups, in proportion to the group’s serving capacity, so each group serves a steady share of the requests throughout training.
The homogeneous samplers run only on B200 with BF16 weights and a BF16 KV cache. They match the trainer’s accelerator and precision, so only kernels and staleness separate them.
Each group of samplers and its share of the requests served.
On Modal, each group of samplers scales up or down as needed, even mid-run, and serves the trainer’s newest weights in its own precision.
Making mixed-sampler RL stable
We start from GRPO without any correction for the mismatch, and add three pieces:
Masked importance sampling (MIS) corrects the weighting. It multiplies each token’s term by its importance ratio and masks tokens whose ratio falls outside a fixed range, so on the tokens it keeps, the expected update follows the training policy and every ratio stays bounded. Without the mask, a few tokens the samplers rarely produce could carry very large ratios and dominate the update.
Score centering (SC) removes drift. At each position, it subtracts the expected gradient of the log-probability under the sampling policy, so in expectation only differences in advantage between tokens move the policy. Importance ratios also remove drift, but masking reintroduces it.
Top-p sampling with mask replay (top-p) keeps rare tokens out and gives both policies the same support. The samplers sample each token only from the top-p candidate set, so rare tokens, which can carry extreme ratios, never enter the data. The trainer then renormalizes its own probabilities over the same set. Runs without it sample from the full vocabulary.
Each sampled token adds its advantage times the gradient of its log-probability to the update. Put together, MIS reweights that term by the importance ratio, SC centers the gradient, and top-p restricts both policies to the same candidates.
SWE-bench Pro V2 pass@4 over training on HARD-51 (left) and the full set (right).
Without correction, training collapses on both setups: on the mixed samplers, GRPO collapses soon after training starts. On the homogeneous samplers, training lasts much longer before ultimately collapsing as well. Each piece of the correction on its own delays the collapse, but every such run ends at or below the base model. Only SC + MIS, with or without top-p, trains stably to the end: pass@4 rises over roughly the first 100–140 steps, then levels off at about 65–75% on HARD-51. The full set shows the same pattern from a higher start, from 89% to about 94%.
Training reward, sampler–trainer KL, trajectory length and episodes ending on repeated format errors, for every run.
Every collapse shares three signs: the training reward falls, the sampler–trainer KL grows 3× or more, and most episodes end on repeated format errors. Trajectory length does not show a consistent pattern: some collapsed runs end their episodes early, while others start repeating themselves and run longer. The stable runs show none of these signs. Matching the trainer’s accelerator and precision only delays the collapse: GRPO on the homogeneous samplers starts from a KL about 4× lower, but follows the same path from about step 100.
A healthy and a collapsed episode, with one cell per turn.
A collapsed episode rarely writes a malformed command. Instead, partway through a task, the model stops calling its tool, and after three replies in a row without a tool call, the agent ends the episode. At GRPO’s step-40 checkpoint on the mixed samplers, 62% of evaluation episodes end this way, a median of 22 turns in, as opposed to 57 turns for passing episodes. The last reply is empty in 98% of them, while only 6% contain a malformed call.
On par with homogeneous samplers at lower cost
Training reward, sampler–trainer KL, gradient norm and entropy for SC + MIS + top-p on the mixed and the homogeneous samplers.
With SC + MIS + top-p, the mixed samplers reach SWE-bench Pro V2 scores on par with the homogeneous samplers’: the two runs’ pass@4 stays close throughout training. Their training reward and gradient norm also follow similar paths, despite a 4.7× larger sampler–trainer KL on average. Entropy matches until about step 160; afterwards, it rises on the homogeneous samplers and stays flat on the mixed ones. Importantly, nothing in how the correction works is specific to this model or these accelerators, so we expect this pattern to hold more broadly.
SWE-bench Pro V2 pass@4 against cumulative sampling cost for SC + MIS + top-p on HARD-51 (left) and the full set (right).
Decoupling the samplers from the trainer also makes these results cheaper to reach. For most of training, the samplers’ cost per update was about 15% lower on the mixed samplers than on the homogeneous samplers. Note that our setup was built to validate the performance of using mixed-precision, mixed-accelerator samplers rather than for minimal cost, so there are more savings to be had: every sampler can be tuned on its own, and the mix can shift toward the most cost-effective samplers.
The next three sections explain these results: where training–inference mismatch comes from, why we don’t use true on-policy RL, and how off-policy correction works.
Where training–inference mismatch comes from
A sampled token can get one probability from the sampler and another from the trainer. The causes span the whole RL system:
SourceExamplesEffect
KernelsSeparate inference and training kernels; no batch invarianceDifferent outputs for the same input, even at the same precision
PrecisionBF16; FP8; MXFP4; NVFP4Formats differ in precision (mantissa bits) and dynamic range (exponent bits, block scaling), so the two sides incur different rounding errors; mismatch can grow as more of the computation is quantized
AcceleratorArchitecture-specific kernels and number formats: no FP8 tensor cores on A100 nor FP4 tensor cores on HopperDifferent mismatch on different accelerators with the same code
Policy stalenessFully asynchronous RL, where samplers run several training steps behind; partial rollouts; in-flight weight updatesMismatch that can grow with staleness
RoutingTop-k expert selection; the sparse-attention indexer’s top-k selectionRouter inconsistency: the two sides select different experts, or different key-value entries, for the same token
Top-p / top-k samplingTokens are sampled only from the top-p or top-k candidatesMismatched action spaces that destabilize training
Multi-turn contextDetokenize–retokenize round trips; chat templates that re-render earlier turns or strip their reasoningThe trainer scores tokens in a context the model never saw