待翻譯：AI at Home Part 2: Multi-GPU Drifting

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

AI At Home Part 2: Multi GPU Drifting No, We Have AI At Home Chapter 2: Multi-GPU Drifting August 20 2026 In the last chapter I built a ridiculous home server out of e-waste-grade GPUs to run AI language models. Here, I'll be talking about what I needed to do to squeeze some kind of reasonable performance out of this kind of setup. (This is going to be using existing code and techniques, messing around with llama.cpp settings and so on; writing new ROCm kernels is out of scope for this chapter). I'm gonna go into some background here. If you already know how transformer models work, go ahead and jump to section 2. If you already know how multi-GPU parallelism works and just want to get to the part where I'm testing things, jump to section 3. section 1: attention Okay, basically all of the AI text generation software that's currently in use everywhere are instances of the "transformer model" or "large language model". The design and basic technique was introduced in the paper Attention is All You Need, which is probably the most important paper in the field of computer science in the past, I dunno, twenty years? The paper is pretty readable as far as these things go. I'm sure every software engineer reading this blog post has already read it, right? (Right?) Anyway. I'm going to over-simplify things a bit here and focus from the perspective of somebody who is trying to get these things to run fast on crappy hardware, and not go deep into the tensor math or talk much about model training, because this blog post is already going to be way, way too long. The LLM works in terms of "tokens". A token is basically a word fragment; instead of inputting and outputting individual letters it's more efficient to chop these up into sequences of letters and have the model process those. (This is why early AI models were bad at correctly answering questions like "how many times is the letter R in the word raspberry?") Different models tokenize language differently, you can think of this as like a frequency encoding. Each time a model generates another token, it'll actually generate a probability distribution and then randomly sample one from that distribution, because language works better that way than picking the exact most likely next thing every time. The language model is a neural network that's divided into layers. You have an input layer and an output layer and a bunch of layers in between that don't directly interact with the input or output ("hidden layers"). The input layer takes in the entire input prompt, and then each layer does math on the output of the previous layer in series. The thing where the model looks at the entire input at once, and looks at the relations between different tokens at different points in the input series, is called the "attention mechanism". If you've been reading about AI language models you probably have heard somebody confidently claim "these AI models are just next word generators, like a Markov chain is", and then you probably noticed that these AI models generate very different outputs than a Markov chain does, and wondered where exactly that guy went wrong. Well, a Markov chain doesn't have the attention mechanism, it just generates a new token based on the previous token in the series. So, to generate the next token, the model reads in the entire tokenized prompt, turns this into an embedding matrix (each token gets turned into a vector where the length is the hidden dimension of each layer), then does the attention math on each layer (gigantic matrix multiplication for each token, for each layer in series), samples a new token, adds it to the prompt, and keeps doing this in a loop until it gets to a token that indicates that it's time to stop. fig 1 from "Attention Is All You Need" (Vaswani et. al., 2017) For our purposes, what this means is that every time the computer generates a token, it needs to read in the existing context, and also every weight in the model, in order to do all those matrix multiplications to generate the token. I'd mentioned the Gemma4-31B model in the last chapter; as the name implies, the model has 31 billion weights (divided into 60 layers). We've got to load all of them into the GPU to calculate the next token. Loading these takes a lot longer than the actual attention math does; token generation is (usually) limited by memory bandwidth rather than compute. This is why the server I built has all of those GPUs with lots of VRAM attached to them; the model weights and KV cache need to be in VRAM that can get to the GPU quickly. The memory attached to the CPU is by comparison a lot slower. (This is also why we are in a memory shortage right now as the entire industry shifts to prioritize producing high-bandwidth memory for data center GPUs). This obviously isn't going to scale super well. As model sizes increase, token generation slows way down; practical limits on this kind of thing got hit already. In response, we have the misleadingly-named "Mixture of Experts" model architecture1. The idea here is that the first couple of layers are used every time to process the whole input embedding matrix, then this gets routed to some subset of the model. For the middle layers, each input token gets routed to a subset of the weights, so only some set of the weights will need to get loaded for each token. Each of these subsets is called an "expert", and I really hate this framing, because it gives you the completely false impression that like one of these branches knows about Python and one of them knows about rocket engines and one of them knows how to speak German and you can just trim the parts of the model you don't care about. This is absolutely not the case, though! The "experts" are basically random, or at least unpredictable, and for most MoE models, different tokens will get routed to different experts in a mostly uniform distribution. So, for example, Deepseek V4 Flash has 284 billion weights but for each token we will only load and process 13 billion. (The shorthand for this is "284B-A13B", only 13B get "activated"). But I don't know which 13 billion ahead of time, and it's going to change for each token that gets generated, so I still need to store all 284 billion weights in some pretty fast memory to keep token generation fast. The MoE thing makes token generation fast at the cost of needing more VRAM. section 2: parallelism When you're using a lot of VRAM, it's going to be divided among separate GPUs. My server has four, with 32GB each; the serious business ones will have eight GPUs with like 192GB each, and then will have to split really big models up between multiple compute nodes, but the principles are the same. A GPU can read its own memory pretty quickly and memory from some other GPU not very quickly and memory on a whole different compute node will be slower still. There's several different ways to split this up. Only two of them really matter for the use case I have, where I've got a box in the garage and I'm the only real user and I'm trying to get the smartest model working at adequate speeds for myself. Layer Parallel - We take the model and put different layers on the different GPUs. As we're generating a new token, we run through a few of the layers on GPU 1, then move that intermediate state over to GPU 2 and process the next set of layers, and so on. This is pretty simple, it gets all of our model weights into VRAM. Each layer is getting processed in series, though, because each layer depends on the output of the previous layer, and so the theoretical best speed we can expect here is basically the same as what one GPU would give us if it had as much VRAM as the whole set. (In practice it'll be a bit slower). An AMD V620 has 512 GB/s of memory bandwidth. If I'm splitting the model up between four GPUs layer parallel, for one prompt I'm gonna get... 512 GB/s of memory bandwidth, minus the overhead of moving data between the GPUs. Tensor Parallel - We take the model and split each layer between multiple GPUs. For each layer, each GPU calculates a fraction of that matrix multiplication, then sends that result to some other GPU that will synchronize and calculate the final result before moving on to the next layer. In theory this should allow us to parallelize our memory bandwidth: I have four cards at 512GB/s, if I'm splitting the model up tensor parallel I should get 2TB/s of memory bandwidth as I move through each layer! Minus the overhead of moving data between the GPUs, of course. Which is, unfortunately, way more! We're moving data around multiple times for each layer, instead of just once per GPU per token as in layer parallel. This can (and will, on my server) outweigh the speed increase from the increased memory bandwidth. You'll also see discussion of other parallelisms that make more sense in the context of serving multiple users at once: Data Parallel is just running the same model on multiple separate GPUs and routing incoming queries to run in parallel. I may play around with that for multiple subagents running simultaneously in the future, but it's not actually necessary until you get to very large scale because a GPU can process inference in a batch already. (The inference pipeline is bound by memory bandwidth, as mentioned earlier, so you can slide some amount of extra matrix math in there for "free", at least until you try to do so much that it becomes compute-bound again). Expert Parallel keeps shared tensors on every GPU and then puts each of the non-shared "experts" (or subgroups of them) onto different GPUs, so multiple users would get their queries routed to different GPUs for most of the inference pipeline. Again, this one mostly makes sense for lots of concurrent users, and is widely used by commercial LLM providers. It doesn't really help my case though. Anyway, my server will mostly be serving me making one request at a time. I'll play around with parallel subagents at some point, maybe I'll have a couple guests hitting this thing at the same time, but I don't really have multiple users here so I'm not optimizing for these cases. section 3: experimentation Okay, let's talk about my server. The GPU array I have here is four AMD Radeon Pro V620 cards connected to a shared PCI Express bus. You'd expect the inter-GPU-traffic to be slow, and it's even slower than you'd expect, because the motherboard is running the older PCIe 3.0 standard and most of the cards are using 8 lanes instead of 16. jacob@daedalus:~$ sudo lspci -vvv [...] LnkCap: Port #2, Speed 16GT/s, Width x16, ASPM L1, Exit Latency L1 lspci shows eight lanes and slow, like the LA freeways So I suspect that tensor parallel isn't going to work very well. The cards that are actually designed and marketed for AI workloads have some kind of high speed, low latency inter-GPU connection in addition to PCIe, and these are all manufacturer-specific; Nvidia has NVLink, AMD has Infinity Fabric, Intel has Xe Link, etc. These cards don't have anything like that. Around the same time AMD was making these cards for their cloud gaming scheme, they were making an AI-focused card called the AMD Instinct MI210, it has about three times the memory bandwidth and you can get a special bridge connector that connects up to four of them via Infinity Fabric. Also, a single one of those cards would cost (used, today) more than this whole box, so that's all kind of a non-starter. I have to make do with what I have. What I have are four cards that are respectably fast individually and have poor interconnects between them, which means we're going to have to do this in layer parallel, then figure out how to fill up the pipeline. And even layer parallel has a performance impact, as we'll see. For the purposes of this study here I had two models I was playing with; I had Gemma4-31B, and Deepseek V4 Flash. Gemma4 31B has 31 billion parameters, at a 4-bit quantization the weights are ab [truncated for AI cost control]