RL post-training on 14 Macs across 4 countries

We just finished a multi-turn reinforcement learning (RL) run of LFM2.5-8B-A1B, [2] an 8.3B-parameter MoE, on 14 Macs spread across four countries. And one of them was my MacBook. The Macs generated the rollouts; a single B200, an ocean away, did the training. On PaperSearchQA,[14] an agentic task where the model answers biomedical questions by searching a corpus of research papers, held-out pass@1 more than doubled, from 29% to 63%. As far as we can tell, this is the first time anything like this has happened: the largest multi-Mac setups we could find, eight EXO machines[9] and four RDMA-linked Mac Studios,[8] all had to be wired to each other. Here there are no wires.

Macs are one of the best hardware choices for post-training. Why? Because most of the compute in post-training is inference, and that's what Macs are good at. In reasoning and agentic RL, generating the rollouts takes most of the run's time, often above 90%.[6][5] We put the gradient step on a datacenter GPU. The rollouts don't need one; they need memory, enough for the model plus the KV cache of dozens of concurrent episodes. Unified memory holds both at sizes no consumer GPU can, and studies comparing Apple silicon against consumer NVIDIA cards keep finding it one of the most cost-effective ways to run large models.[21][22] MoEs suit it especially well: big in memory, light in compute.

However, this is not an easy system to build. Two problems stand in the way. First, the public internet: with no wires, every worker reaches the trainer over an ordinary internet connection. Those connections are drastically slow compared to a local interconnect, and the machines behind them come and go mid-run. Second, the off-policy gap. Every RL post-training system has one: the inference engine and the trainer never match exactly,[7][10] but the mismatch is usually mild. Ours is drastic: different hardware (Apple silicon vs an NVIDIA B200), a different kernel stack (MLX vs Megatron), a different precision (int8 vs bf16), and weights a few versions stale on top. The standard RL systems don't work in this setting.

In our setup:

To the best of our knowledge, this is the first RL post-training run that used only consumer Macs for rollouts. We use fourteen mixed-generation Macs across four countries, connected over ordinary internet.

The trainer trains while the Macs generate asynchronously. Nothing goes idle.

Rollouts run in int8 on Apple's MLX;[3] the trainer runs bf16 on NVIDIA's Megatron.[4]

Workers join, leave, or crash mid-run without stalling the run.

Nothing caps the fleet size; we ran 14 Macs only because that saturated one B200.

Each weight update ships about 110x smaller than a full checkpoint.

Rollouts stream to the trainer as they finish, not at sweep end.

Custom Metal paged-attention kernels, specialized for the LFM2.5 architecture.

DPPO's token gate keeps training stable across a drastic off-policy gap.

Held-out pass@1 more than doubles, from 29% to 63%.

The full stack is open source.

The rest of this post explains how we made each piece work.

1 · What's the right topology for a fleet you can't wire?#

We begin with the topology. The communication pattern of the system is minimal: each Mac needs the newest weights, the trainer needs the generated rollouts, and no worker ever needs data from another worker. The fleet adds its own constraints: consumer Macs sit behind NAT, so no node can be required to accept an inbound connection; machines join and leave during a run; home connections vary wildly in speed and stall without warning, so nothing can run in lockstep. On the training side we assume the trainer runs within a centralized cluster, which makes that cluster the one source of truth for the weights. By this point the traffic is clearly shaped like a star (weights fan out one-to-many, rollouts gather many-to-one, and no edge connects two workers), so we chose a star as our topology; Figure 1 shows this topology.

Latency forced a second decision. The trainer and the Macs are disaggregated, an ocean and a consumer uplink apart, so every transfer between them carries real delay no matter how much we compress. A synchronous loop would spend most of its time waiting on that delay. So we made the decision to make the system fully asynchronous: the trainer stores arriving rollouts in a replay buffer and trains from it, each Mac generates from whatever weights it has, and nobody ever waits for anybody. Most of the design decisions in this post follow from that choice, and its price, the off-policy gap, gets a section of its own.

The hub is a storage bucket: plain S3-style object storage. The trainer writes each new version of the weights to the bucket and every Mac reads it back; generated rollouts flow the other way, one object per group. That's what the animation at the top of the post is showing: gold packets are rollouts converging on the trainer, navy packets are weight updates spreading back out to the Macs.

No machine ever has to accept an inbound connection, because everything is an outbound HTTPS request against storage.

Objects are only ever added, never edited in place, so a retried upload lands whole or not at all, and a single small pointer file names the newest version of the weights: a Mac joining mid-run reads the pointer, downloads the latest weights, and starts generating.

For storage we chose Cloudflare R2: the access pattern is write once, read by everyone, so egress dominates the bill (about 230 GB of weight pulls over this run alone, growing with every worker), and R2 charges nothing for egress.[18]

Still, the topology only decides where the bytes go, not how fast they get there. The star has to move ~9 GB of weights (the 8.3B-parameter model in int8, quantization scales included) out to every Mac every several minutes, and a constant stream of rollouts back. Sent the naive way, both arrive stale, and stale is exactly what destabilizes training. The next two sections discuss how we make this connection fast.

Figure 1: The star: every node talks only to the bucket. Rollouts flow up in gold, weight deltas flow down in navy. No worker ever opens a connection to another worker or to the trainer.

2 · Weight sync: home internet at datacenter speed#

We start with the weights, the largest payload the system moves. The trainer publishes a new version of the model roughly every ten minutes, and the int8 weights are 9 GB: at the global median speed, roughly 100 Mbps,[25] that download takes about ten minutes, the entire gap between versions. While it runs, the system has two options, and both are expensive. Either the trainer waits for rollouts generated on the new version, which leaves the one datacenter GPU in the system idle, or it keeps training on rollouts from weights that are now several versions old, which widens the off-policy gap that can destabilize a run. We wanted weight-sync to be so efficient that we never have to make either sacrifice.

To make this happen we use PULSE, a weight-sync technique that cuts what has to be sent by roughly a hundredfold.[13] It starts from a precision mismatch. The trainer keeps its master weights in fp32, and every optimizer step moves them by a tiny amount, on the order of the learning rate. The Macs never see that precision: rollout inference runs in int8, where a weight can only take one of 256 evenly spaced values per quantization group. Those few representable values form a coarse grid (Figure 2), and a weight's int8 value moves only when its accumulated fp32 movement crosses over to the next point on the grid. At RL learning rates (e.g., 10−6) almost no single update is that large, so the grid absorbs nearly all of them, and the int8 view of most weights is identical from one version to the next.

The trainer therefore ships only the weights whose int8 value changed, and the Mac reconstructs, bit for bit, the same int8 weights a full download would have given it. The PULSE paper works the mechanism out in detail: about 99% of per-step updates are invisible even at bf16, and the coarser the inference precision, the more the grid absorbs, so the cast that saves the Mac memory also buys the compression.[13]

Figure 2: The grid absorbs the update. A stored int8 weight can only take the values on its grid, so it moves only when the trainer's underlying value crosses a grid step. Most updates never cross one; the few weights that do are the whole delta, and shipping them rebuilds the new weights bit for bit. The animation compresses time; in practice about 99% of updates are absorbed. What this bought on the live run is measured in the results section.

Lower precision buys more compression, not less: our coarse int8 grid hides even more of each update than the bf16 the PULSE paper measured.

At a hundredfold, the 9 GB download becomes about 90 MB, and 90 MB over that median connection lands in about seven seconds, the same wall-clock as pulling the full 9 GB over a 10 Gbps link. For the heaviest traffic in the system, a home connection behaves similar to a datacenter connection. And our live run confirms it: a new version reached the bucket in seconds, not minutes.

Two more design decisions round out the sync, both taken from the PULSE paper, along with its recommendations for encoding the patches.[13] First, every delta is computed against the previous version, not against the last full snapshot. Diff against a fixed snapshot and the weight delta can only grow: weights keep crossing grid steps as training moves, so a few dozen versions out, each delta would carry a large fraction of the model. Diff against the previous version and each delta carries only the weights that crossed in that one publish interval, so deltas stay the same size no matter how long the run gets.

The cost is that a fresh Mac would have to replay every delta since the start, and that's the second decision: the trainer also publishes a full int8 anchor every twentieth version, purely as a starting point (Figure 3). A Mac that joins mid-run, or falls behind, downloads the newest anchor and replays the few deltas since the anchor, not the whole chain. And the anchor is int8 rather than bf16 on purpose: the download drops from 16.9 GB to 9 GB, and the weights sit at about 8.4 GB in memory, which is what makes a 16 GB personal MacBook viable as a worker.

Figure 3: Joining is an anchor plus a short tail. Deltas chain version to version, and every twentieth version the trainer publishes a full int8 anchor. A Mac that joins at v27 downloads the v20 anchor and replays seven deltas, never the whole chain. On the live run, a cold join went from nothing to serving current weights in 124 seconds.

3 · Rollout streaming: no group waits for the sweep#

Weights are half the traffic. The other half flows the opposite way: generated rollouts. Each Mac works in sweeps: one sweep is all eight of its groups (eight on an M4 Pro; a smaller Mac runs fewer), eight rollouts each, decoded together as one batched pass, with episodes evicted as they finish so they stop consuming compute.1

Shipping is decoupled from decoding in two places. First, uploads run on a background thread, so the decode loop never blocks on the network; we measured no decode-speed loss. Second, a group ships the moment its last episode finishes rather than when the whole sweep ends. As Figure 4 shows, the trainer starts consuming a Mac's first groups while that Mac is still decoding its last, and a slow episode delays only its own group. Rollouts arrive at the bucket as a steady stream rather than periodic bursts.

Figure 4 lays the whole system on one timeline, the ten minutes between one weight publish and the next: the Mac pulls a delta and runs its sweep, the groups land in the bucket as they finish, and the trainer fits four optimizer steps and publishes again. Nothing blocks anything else. Next section discusses how wide the sweep should be.

Figure 4: One publish inter

[truncated for AI cost control]