NVLink, NVSwitch, and All That

I wrote before about ‘scale-out’ fabrics — InfiniBand, RoCE, and more generally RDMA — the networks and paradigms that stitch thousands of machines into a datacenter island.

This post is about the other kind. The names come from an old argument about servers: when you need more computer, you can either buy more machines and scale out, or buy a bigger machine and scale up. A scale-up fabric is an attempt to connect processors so tightly that they can behave as one. Inside a modern (NVIDIA) GPU machine the fabric is NVLinkThe terms are a bit frustrating really. Because you can scale up your LLM inference domain over your scale-out network. And you can certainly scale out inside a scale-up domain..

Fast, or far

Let’s follow the physics of sending bytes over a link. At the lowest level: an NVLink is a bundle of wires, set up as differential pairs. Each pair is driven by a SerDes: a circuit that turns wide, slow on-chip data into one very fast signal on the wire.

The signal on the wire is just a voltage changing over time. To turn it back into zeros and ones the receiver has to sample it at the right instants. Before deciding what to put on the wire, we have to decide where the receiver’s sense of time is going to come from.

Length-matching meanders next to a BGA package (source).

One answer is to send a clock pulse, separate to the data, on its own wire, and sample as the clock ticks (a forwarded clock). This is how DDR memory and HBM work. But it only works if the clock’s ticks arrive close enough to all the signal bits for the receiver to sample them cleanly. The receiver can deskew small differences, but every data wire still needs about the same delay and, in practice, about the same length. This is why PCBs are covered in squiggles. At a hundred gigabits per second, one bit occupies about ten picoseconds: a millimetre or two of trace. That kind of matching is possible across centimetres, and gets much harder across metres.

We can instead synchronize each wire individually, sending its clock over each signal wire itself. We know that every transition in the signal (each 0-to-1 or 1-to-0) marks a bit boundary. So we can use a circuit at the receiver (clock and data recovery) that keeps a local oscillator locked to those transitions. The problem is that it works only if the transitions keep coming. If you send a million zeros in a row, the oscillator can start to drift. To keep the data transitioningStatistically, not provably. In principle you can craft data that XORs to a long run of zeros ("killer packets", seen in SONET). Modern links use long scrambler polynomials to make this impractical., we XOR it with a pseudo-random sequence that both ends can generate (scrambling).

Top: each edge in the data re-centres the receiver's clock, but across the run of zeros it free-runs (drawn a few percent fast) and the samples slide off-centre until one lands in the wrong bit. Bottom: the same payload scrambled, the edges never stop.

Once we can carry a timing signal, we need to put our 1s and 0s on the wire. One of the simplest ways is: high voltage for a 1, low for a 0. That scheme is called NRZNon-return-to-zero: return-to-zero being a version where you return to zero between every signalled bit.

To signal faster on a single wire under NRZ you have to tick faster. A trace on a board works like a low-pass filter: the copper, and the insulation around it, both absorb more at higher frequenciesThe copper loses because high-frequency current crowds into a thin shell at the conductor's surface (the skin effect). The insulation loses because the signal's energy travels through it via the electromagnetic field, and the material's molecules absorb some of it as the field alternates (dielectric loss)., adding attenuation that the receiver has to make back. It does this with filters that re-boost the high end, but this boosting amplifies the high-frequency noise along with the signal.

Rather than keep paying that rising attenuation, at some point it’s better to try to send more bits per tick. PAM4 signals four voltage levels instead of two, so each tick carries 2 bits of information. The cost is that the levels sit a third as far apart. As the level spacing gets closer, errors in decoding start to become more likely.

PAM4 links aim for roughly one bit in ten thousand wrong before correction, where NRZ links without correction can aim for better than one in a trillion. At these speeds an error rate of one in a trillion means an error every ten seconds or so, which is rare enough you can manage it at the link layer with checksums and resends. One in ten thousand means millions of errors a second, which is far too many to handle by resending. So PAM4 is usually paired with forward error correction. This costs latency, since the decoder has to collect a whole block before it can correct anything.

Instead of signalling furiously on a single wire, can’t you just add more wires? Yes, up to a point. Fast signals have to leave from the die’s edge: a SerDes buried mid-die would have to thread its traces out past all the stuff around it. Off-chip bandwidth therefore scales with the die’s perimeter (its shoreline) while compute scales with its area. The reticle limit caps the die at roughly 26 by 33 mm: about 120 mm of shoreline. The HBM interfaces, NVLink, PCIe, and C2C all compete for that finite edge.

The other variable is distance. Higher frequency signals travel less far: passive copper carries 100 Gb/s per pair about two metres, and 200 Gb/s about oneThese are IEEE spec objectives: IEEE 802.3ck specifies 100G-per-lane passive copper to at least 2 m; the follow-on P802.3dj carries a 1 m objective at 200G per lane, and TE and Intel demoed 224G PAM4 over a 1 m passive cable at OFC 2023. Longer is possible with effort: Marvell has driven 2.5 m of cumulative copper with a long-reach 224G SerDes.. To go further requires extra hardware steps. A redriver is an analog amplifier, and a retimer a kind of digital one: a full receiver and transmitter back-to-back decodes the bits and relaunches the signal anew.

The ledger for all this stuff is energy per bit transferred. Between chips on the same board, the channel is clean enough for many slow, simple wires: NVLink-C2C, the link between a Grace CPU and its Hopper GPU, has a forwarded clock, and moves 900 GB/s for 1.3 pJ/bit. NVLink between GPUs uses the SerDes treatment described above, at around 4 pJ/bit: at Blackwell’s 1.8 TB/s of NVLink bandwidth, roughly 58 W per GPU spent moving bits off the package. Past passive copper, a retimer at each cable end takes it to around ten, and optics to nearer twenty (the price of optics is the conversion from electrons to light, and is mostly indifferent to the distance travelled)1.

Energy per bit against reach, one bar per link technology. These are quite fuzzy, but the direction is correct.

The binding constraint is where that energy cost is paid. Every joule the links expend is a joule inside the rack’s power and cooling envelope, which competes with compute. For now that means passive copper, and therefore scale-up stays within a rack.

How we got here & where we’re going

Reading another chip’s memory with an ordinary load is an old idea for CPUs. SGI’s NUMAlink stitched as many as 512 processors into a single memory space by 2001, and coherent HyperTransport did the same for the handful of sockets on an Opteron board. Commodity clusters killed the big machines, but the board-scale version is everywhere: every multi-socket server is still a small NUMA machine.

On the GPU side, NVIDIA added a single virtual address space across GPUs, with peer-to-peer access over PCIe, in CUDA 4.0 (2011)CUDA 4.0 introduced unified virtual addressing and GPUDirect peer-to-peer on Fermi. Whether two GPUs can map each other's memory depends on the bus between them; over PCIe it also requires both devices to sit under the same root complex.. By 2016 a gen3 x16 slot moved 16 GB/s per direction against 720 GB/s of local HBM2: one forty-fifth the bandwidth. NVLink was introduced with the P100 that year to try to widen the path: four links per GPU, with 160 GB/s of aggregate bidirectional bandwidth. The first generation also joined CPUs to GPUs: IBM sold POWER8 servers with the CPUs wired to P100s over NVLink.

Four links can’t fully connect eight GPUs, so the DGX-1 wired its P100s into a hybrid cube mesh, each GPU directly linked to four of the others. Transfers took one hop or two depending on the pair, and NCCL threaded its rings around the physical wiring. In 2018 NVIDIA put a switch between the GPUs instead: NVSwitch, an 18-port chip that carries traffic between all of its ports at the full link rate simultaneously. The DGX-2 holds twelve of them, and any pair of its sixteen V100s can transfer at the full 300 GB/s, with all sixteen GPUs communicating at once.

The third-generation switch (with Hopper, in 2022) carried ALUs for in-network reductionNVIDIA's name for this is SHARP, the Scalable Hierarchical Aggregation and Reduction Protocol. It came over with the 2020 Mellanox acquisition: Mellanox's InfiniBand switches had been doing in-network reductions for MPI collectives since 2016.: in an allreduce, every GPU sends its data into the switch once and receives the sum once, instead of shuttling partial results around a ring. That halves the bytes each GPU moves, and roughly doubles the effective bandwidth of the collective.

The GH200, in production since mid-2023, brings the CPU into the domain. It’s a Grace CPU paired with a Hopper GPU over NVLink-C2C, with the CPU’s 120 GB of LPDDR5X and the GPU’s 96 GB of HBM in one address space.

The GB200 followed in 2024, with one Grace for every two Blackwell GPUs. The NVL72 is 36 of them wired into a rack, the 72 GPUs and 36 CPUs joined through eighteen switch chips and a copper spine, 130 TB/s of any-to-any bandwidth between them. NVIDIA calls the rack “a single, massive GPU”. A thread on any of the 72 can issue a load against any byte of its 13.4 TB of HBM.

The link generations so farPair counts are per direction; per-GPU figures are NVIDIA's bidirectional aggregates.:

genGPUyearper pairpairs per linklinksper GPU

1P100201620 Gb/s NRZ84160 GB/s

2V100201725 Gb/s NRZ86300 GB/s

3A100202050 Gb/s NRZ412600 GB/s

4H1002022100 Gb/s PAM4218900 GB/s

5B2002024200 Gb/s PAM42181,800 GB/s

NVIDIA has also named the next three stepsFrom NVIDIA's Vera Rubin material and its GTC 2026 roadmap. Blank cells are numbers NVIDIA hasn't yet put against the names.:

NVLinksystempromisedper pairper GPUdomain

6Vera Rubin NVL722026200 Gb/s PAM4, both directions3.6 TB/s72 GPUs, one rack

7Rubin Ultra NVL5762027——576 GPUs, eight racks

8Feynman2028——1,152 GPUs, eight Kyber racks, optical between racks

Vera Rubin keeps the rack as it is — 72 GPUs, a spine of five thousand copper cables — and doubles the bandwidth through it, to 260 TB/s in aggregate. The tick rate and the modulation are unchanged from NVLink 5, and the spine has no room for more cables. The doubling comes from each pair carrying both directions at once. Both ends transmit onto the same two wires at the same time, so the voltage at each receiver is the sum of two signals, one of which, its own, the receiver already knows. It synthesizes a copy of what its transmitter is sending and subtracts it; what remains is the far end’s signal. The subtraction is a timing problem: a tick lasts under ten picoseconds, and the copy has to line up with the actual echo to within a fraction of that, or the residue swamps the far end’s signalFrom SemiAnalysis's Vera Rubin analysis; NVIDIA's own material stops at the aggregate numbers. Gigabit Ethernet has driven its four twisted pairs this way since 1999, at a tick rate nearly a thousand times slower..

The NVL576 is the first NVLink domain bigger than a single rack: eight of them, with copper and optical links between the racksNVIDIA's words: eight MGX NVL racks

[truncated for AI cost control]