Understanding Go AI Inference: What Is Inference?

"Welcome to a new series! For most developers today, using a large language model means one thing: an HTTP call to somebody else\u0026rsquo;s computer. You send a prompt to an API, tokens come back, and everything in between is somebody else\u0026rsquo;s magic.\nBut here\u0026rsquo;s what I find much more interesting: you can run these models locally, inside your own process, on your own hardware — and if you\u0026rsquo;re a Go developer, you can do it directly from Go. Two projects make this genuinely pleasant today: Yzma, which lets Go call the llama.cpp libraries directly (without cgo — that gets its own article later in the series), and Kronk, which builds a high-level, OpenAI-API-feeling SDK and model server on top of Yzma. And underneath both of them sits llama.cpp, the C/C++ inference engine that made running LLMs on ordinary hardware practical.\nIn this series we\u0026rsquo;re going to understand that whole stack from the inside: what llama.cpp actually does when it \u0026ldquo;runs a model,\u0026rdquo; how Yzma manages to call it from Go, and how Kronk turns all of that into an engine you\u0026rsquo;d actually want in production.\n📌 A note on scope\nThe goal of this article is to understand the mechanics around the model — how a model file is stored, loaded, and run — not what happens inside the neural network itself. The math inside those layers is a whole topic of its own, and it\u0026rsquo;s out of scope here.\nThe llama.cpp details are against master as of July 2026 (commit ad8d8219) — llama.cpp moves fast and doesn\u0026rsquo;t do stable releases, so I\u0026rsquo;m pinning a commit instead of a version. And this is deliberately a conceptual tour: we\u0026rsquo;re not going to get into the C++ code details. We\u0026rsquo;ll also stay out of the Go layers entirely — Yzma and Kronk get their own articles.\nWith the scope set, let\u0026rsquo;s dive in.\nWhat Inference Actually Is Let\u0026rsquo;s start by demystifying the word. A language model has two very different lives.\nThe first life is training: you show the model terabytes of text, and an optimization process gradually adjusts billions of numbers — the weights (also called parameters) — until the model gets good at one narrow task: given a sequence of text, predict what comes next. Training takes months and data centers. We\u0026rsquo;re not talking about training in this series, at all.\nThe second life is inference: training is over, the weights are frozen, and now you just\u0026hellip; use them. The best mental model is a plain, pure function: it takes two inputs — your sequence of text and those billions of frozen weights — and returns one output, a prediction for the next token. Something like nextToken(input, weights). That\u0026rsquo;s it. That\u0026rsquo;s the whole secret. No learning happens, nothing is updated — the weights are read-only arguments, so a model file is exactly as smart the millionth time you call it as the first.\n📌 A note on weights\nIf you\u0026rsquo;re curious about how much memory those weights actually take up — how quantization shrinks them, and why they usually dominate the VRAM you need to run a model locally — Kronk\u0026rsquo;s VRAM calculator write-up is a nice read.\nSo when you download a \u0026ldquo;model,\u0026rdquo; what you\u0026rsquo;re downloading is essentially a giant pile of frozen numbers — those weights — plus a description of how to use them. And \u0026ldquo;running\u0026rdquo; the model means doing a very large amount of arithmetic between your input and those numbers. If that sounds suspiciously hand-wavy, don\u0026rsquo;t worry: we\u0026rsquo;ll demystify the weights later in the article, when we crack open an actual model file. For now, all you need to hold on to is that a model is a huge pile of numbers that somebody else already calculated for you.\nWe\u0026rsquo;ve already mentioned tokens a couple of times now — but what exactly is a token? Let\u0026rsquo;s see.\nTokens: the model\u0026rsquo;s alphabet There\u0026rsquo;s one catch before any arithmetic can happen: models don\u0026rsquo;t read text. They work with tokens — integer IDs from a fixed vocabulary, where each ID corresponds to a chunk of text (a word, a piece of a word, a punctuation mark).\nOur prompt \u0026ldquo;The capital of France is \u0026quot; might become something like five or six token IDs — say [791, 6864, 315, 9822, 374] (the exact IDs depend entirely on the model\u0026rsquo;s vocabulary). From this point on, the model never sees letters again; everything downstream is done in terms of these integers.\nSo the model receives a list of integers. What does it actually do with them?\nOne pass, one prediction Here\u0026rsquo;s the shape of the computation, stripped to its essence. One important thing up front: the model doesn\u0026rsquo;t process the input word by word in separate rounds — the whole sequence of tokens goes through the network together, in a single pass, and out comes a single prediction. Let\u0026rsquo;s walk through it with this diagram:\nReading it left to right, we start by converting each token ID into an embedding — a vector of numbers that represents the token in a form the model can work with (if you\u0026rsquo;d like to understand embeddings better, Weaviate has a nice primer). Then all of those vectors are fed into the model together, which runs a ton of calculations and ends up producing a list of scores — one per token in the vocabulary — saying how likely each token is to come next. And from that list, we pick one: \u0026quot; Paris\u0026rdquo;.\nThat final pick has a name — sampling — and we\u0026rsquo;ll come back to how it really works later. But notice something crucial first: all this work produced just one token. How do we get a whole sentence out of a machine that only predicts the next word?\nThe autoregressive loop By doing it again. And again. This is the autoregressive part, and it\u0026rsquo;s what makes an LLM conversation tick. Let\u0026rsquo;s see how it works with this diagram:\nFollow the arrows. We feed our sequence — \u0026ldquo;The capital of France is \u0026quot; — into the model, it does one pass, and we pick a token: \u0026quot; Paris\u0026rdquo;. Here\u0026rsquo;s the trick that turns a single prediction into a whole answer: instead of stopping, we append \u0026quot; Paris\u0026quot; to the sequence and run the model again, now on \u0026ldquo;The capital of France is Paris\u0026rdquo;. The next token might be \u0026ldquo;.\u0026rdquo;, we append that too, and around we go — one new token per lap — until the model picks a special end-of-generation token (its way of saying \u0026ldquo;I\u0026rsquo;m done\u0026rdquo;), or we hit a length limit and stop it ourselves.\nEvery word of every LLM answer you\u0026rsquo;ve ever read was produced one token at a time by this loop. Generation isn\u0026rsquo;t one big computation — it\u0026rsquo;s the same next-token prediction, in a for loop.\nSo much for the theory. Before we can watch that loop actually run, everything hinges on one thing we\u0026rsquo;ve been hand-waving about: those frozen weights. Where do they come from, and what does a \u0026ldquo;model\u0026rdquo; actually look like on disk? Time to open the box.\nGGUF: The Whole Model in One File Time to demystify those weights. That pile of frozen numbers isn\u0026rsquo;t a shapeless heap: it\u0026rsquo;s organized into named arrays called tensors (a tensor is just an n-dimensional array — a matrix generalized to more dimensions) — for example, a single layer of the neural network is typically made up of several tensors. GGUF (GGML Universal File) is how that bag of tensors is stored on disk, and its whole design follows from one goal: a single, self-describing file. One .gguf holds everything needed to run the model — weights, architecture, hyperparameters, the entire tokenizer, even the chat template — with no sidecar files to download. Copy one file, run the model. (The very largest models sometimes get split into a few .gguf shards for practicality, but the principle holds.)\nSo what does one of these files actually look like inside? The format is documented right at the top of the GGUF header in ggml — the tensor library llama.cpp is built on, and where the file format itself lives (ggml/include/gguf.h). But rather than paste the raw spec, let me draw it for you:\nLet\u0026rsquo;s read it top to bottom. Right at the start sits a tiny header — the magic bytes GGUF (so a program can check \u0026ldquo;yes, this really is a GGUF file\u0026rdquo;), a version number, and two counts telling the reader how many tensors and how many metadata entries to expect. Everything after that falls into three sections, and the rest of this section is really just a guided tour of those three boxes: metadata, tensor descriptors, and the data blob. Let\u0026rsquo;s take them one at a time.\nThe metadata: a model that describes itself That\u0026rsquo;s box 1 in the diagram. The metadata is a flat list of typed key-value pairs — the values are simple scalars, strings, and homogeneous arrays. The keys are namespaced strings, and there are a few namespaces (the ones you can spot in the diagram) I want to call your attention to:\ngeneral.* — identity: general.architecture says which kind of model this is (llama, qwen2, gemma, \u0026hellip;). This one key decides which graph-building code llama.cpp will use to set up the inference machinery. \u0026lt;arch\u0026gt;.* — hyperparameters, prefixed by the architecture name: llama.embedding_length (the size of those token vectors), llama.block_count (how many layers), llama.context_length (how many tokens fit in the window), etcetera\u0026hellip; tokenizer.* — the entire tokenizer (the piece responsible for turning our text into tokens) lives inside the model file. tokenizer.ggml.tokens is the full vocabulary as a string array, alongside the tokenizer\u0026rsquo;s configuration parameters and even tokenizer.chat_template — the template that turns a chat conversation into a flat prompt string. When llama.cpp opens a file, it reads general.architecture first, and everything else follows from there.\nThat\u0026rsquo;s the what of the model. Now let\u0026rsquo;s see where the actual numbers live.\nThe blob: getting the weights into memory Now for boxes 2 and 3 in the diagram, which work as a pair. Box 2 is the tensor descriptors: one row per tensor with its name, shape, and type — but no data. Instead, each row carries an offset, and that\u0026rsquo;s the red arrow in the picture: the offset points at where that tensor\u0026rsquo;s bytes actually live inside box 3, the data blob.\nThe interesting part is how those bytes get into memory. The loader (llama_model_loader, src/llama-model-loader.cpp) first parses only the header, metadata, and tensor descriptors — no tensor data is read at all. That\u0026rsquo;s enough to know what goes where: some layers stay on the CPU, others go to the GPU, and each destination handles its bytes its own way. For CPU tensors, llama.cpp by default doesn\u0026rsquo;t read() gigabytes of weights into freshly allocated buffers — it maps the file into the process\u0026rsquo;s address space with mmap and points each tensor directly at mapping_address + tensor_offset. For GPU tensors that trick doesn\u0026rsquo;t work — VRAM is a different memory altogether — so their bytes really do get copied: uploaded from the file into GPU buffers, once, at load time.\nSo now we have the file\u0026rsquo;s tensors sitting in memory, each in its right place. Time to run something.\nThe KV Cache: Why Generation Stays Fast Let\u0026rsquo;s start with what we already have: our model\u0026rsquo;s data, the part that never changes. Loading it hands us the llama_model — the file itself, in memory: the weights, the tokenizer, all the read-only stuff. Because nothing about it changes, it can be shared — one loaded model, many conversations, and those multi-gigabyte weights sit

[truncated for AI cost control]