翻訳待ち：Introduction to Post-training

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

This is the first article in a series about post-training. Follow along on Radar. Before post-training, there was a major problem with LLMs: Almost nobody could use them. The story of post-training is also the story of how AI went from a research curiosity to a product used by about a billion people. Post-training is the reason why a model behaves a certain way. This set of training techniques makes LLMs useful (e.g., able to chat with people and interact with AI agents), safe (e.g., aligned with human intentions), and more capable (e.g., through “reasoning” to tackle difficult tasks). Behavior is powerful, and doesn’t just mean holding a conversation or following a user’s instructions. Behavior includes making it possible for the model to use tools, like a calculator tool, a search API, or any application through an MCP. Behavior can even elevate a model’s intelligence, for example by teaching the model to use “reasoning”: that is, working through problems before giving a final answer rather than “guessing” or “memorizing.” From GPT-3 to ChatGPT: The post-training revolution GPT-3 showed up in June 2020. A completion engine, it followed patterns it had seen from its pretraining data, which were not predominantly chat conversations. Imagine scraping data on the internet: that pretraining data had a lot of questions that were followed by other questions—for example, on an exam template. GPT-3 was 175B parameters, large for its time, and it had a wide, general range of abilities, although many of them were latent. If you gave GPT-3 a prompt like “Why do people like golden retrievers?” it might say something nonsensical: Why do people like labrador retrievers? Why do people like poodles? 10 Reasons You Should Adopt a Dog Today These answers look absurd in isolation, but if you imagine a web page with a list of FAQ links, this is a perfectly reasonable next chunk of text. GPT-3 might have just been completing a listicle on a website, because it had seen millions of websites in its pretraining data. The common way to nudge GPT-3 to answer a question back then was by prompt engineering with a Q&A template and few-shot examples. Q: Why do people like labrador retrievers? A: Because they are friendly, loyal, and easy to train. Q: Why do people like beagles? A: Because they are curious, great with kids, and have a gentle temperament. Q: Why do people like golden retrievers? A: Then, GPT-3 might say: Because they are affectionate, patient, and make excellent family pets. While this technique worked, it was brittle. If you forgot the few-shot examples, rephrased the question, or even added a space after “A:,” you’d get something completely different (possibly unhinged) that was far from a reasonable response. In fact, if you were a researcher working with GPT-3 at the time, you probably at some point found the space at the beginning of the response ” Because they are gentle dogs.” annoying and would try to end your prompt with a space “A: ” instead of “A:”. In those cases, it was common for GPT-3 to go off a cliff and produce a drastically different response, sometimes completely off like “dogs dogs dogs dogs…” repeating indefinitely. The reason behind the differing responses to “A:” and “A: ” is because “A:” might tokenize to one token while “A: ” tokenizes to two different tokens. The model literally sees different input sequences, each with different statistical completions in its training data. It’s like asking two completely different questions. While a space is a tiny syntactic change that is meaningless to a person, it becomes extremely meaningful to the model that now sees two different prompts (the tokens change!) with two very different statistical futures to complete. You still encounter the modern equivalent of this when working with chat templates. If you forget to apply the model’s chat template and instead just concatenate 'User: ' + prompt + '\nAssistant: ', you’re sending the model a token sequence that it was not robustly trained on. The tokens are wrong, not the model. Post-training teaches the model to respond to specific token patterns (like user\n in Qwen models). Not using them is like speaking to someone in a language they half-understand. However, most open source models will be trained to be at least somewhat robust without their templates too. Under those circumstances, most people would assume AI still didn’t work. The model wasn’t trained to answer questions; its data wasn’t primarily conversation transcripts. Instead, it was trained to predict the next token in downloaded websites, articles, and documents. Thankfully, this can all be fixed with post-training. And that’s when most people started to believe that AI had undergone a paradigm shift and just might work. Post-training versus pretraining Pretraining heavily influences the model’s knowledge capacity prior to post-training. The model gets raw intelligence during pretraining. Then, during post-training, that intelligence is made useful through behaviors like dialogue and reasoning. In a frontier lab, these two phases are such different processes that very different teams work on them. A model’s factual knowledge about the French Revolution, its understanding of Python syntax, and its grasp of calculus all come from pretraining. Post-training primarily shapes which knowledge the model reaches for, how it presents that knowledge, what tone it uses, whether it declines certain requests, and whether it thinks step-by-step before answering, though targeted SFT on new domains can introduce information the model didn’t encounter in pretraining. If a model gives a wrong answer about history, the root cause is likely in pretraining data, but the practical fix might still come through post-training—for example, teaching the model to use search tools, express uncertainty, or chain-of-thought verify its own claims. But if a model gives correct information in a condescending way or refuses to help with a reasonable request or fails to use tools when it should, those are squarely post-training problems. Pretraining The work of pretraining is centered around cleaning and curating large-scale data, optimizing the model toward relatively clear loss signals, and working with scaling laws given bounded compute. In pretraining, the model learns to predict the next token across a large curated dataset, typically for one or a small number of passes over the training data, though some models train for multiple epochs, especially as high-quality data becomes scarce relative to compute budgets. This is where you’ll hear how a model is fed the entire internet’s worth of data to gain intelligence, although in practice nearly all of the data (often 90% or more) may be thrown out because it’s unsuitable for training. Pretraining is an unsupervised process that runs at increasingly larger scales to match the size of the model. While scaling, thousands of experiments are used to understand what data mix, what architecture considerations, what compute optimizations, what hyperparameters can lead to the best results. There’s variance in each run due to stochasticity found in both software and hardware, so multiple experiments are needed to verify results. Because compute is limited and needs to be used sparingly, researchers will scale iteratively, expanding to the next, say, 10x compute budget, when they gain confidence in the right configuration. A full run isn’t possible to iterate on due to the compute cost and time it would take: The final run, often called the “god run,” can take over a month on thousands of GPUs. Pretraining progress is typically very clearly measurable, using a metric like perplexity, which measures, roughly, the model’s average uncertainty per token. Lower is better, where 1 means the model knows with absolute certainty what token comes next. Meanwhile, a perplexity of 50 means the model’s predictions are, on average, as uncertain as if it were choosing uniformly among 50 equally likely tokens—though in practice, the distribution is peaked, not uniform. Post-training Rather than consuming hundreds of millions of tokens of internet data, post-training operates on far more intentional datasets for downstream tasks. These datasets include human-written demonstrations of ideal responses, human judgments about which responses from the model are better, and carefully designed functions that score the model’s outputs programmatically. They shape what “good” looks like. Like pretraining, post-training can also be more effective with scaling data and compute. Specifically, massive compute budgets have been dedicated to post-training to learn reasoning capabilities (or the ability for models to “think step-by-step” to arrive at more logically sound answers), matching the scale of pretraining compute. Post-training is messier than petraining, which has an elegant, clear optimization objective to minimize the loss over the next token prediction across a huge corpus. The goals of post-training are things like “be more helpful” or “don’t say harmful things.” Many of these objectives are inherently subjective and require human judgment, proxy models that approximate human judgment, or programmatic verifiers that can become elaborate or inefficient. The loss curves are noisier. The quality of the data and feedback matter even more. The scale of post-training is also more complicated than in pretraining. Standard post-training remains relatively modest in compute: tens to hundreds of GPUs for days rather than thousands of GPUs for months needed in pretraining. This makes post-training for alignment highly amenable to rapid iteration; researchers can try something, observe results, form a hypothesis, and run again on a timescale of days. The picture changes dramatically when post-training is used to develop reasoning capabilities. For reasoning models, the compute dedicated to post-training can easily account for half of the overall compute of the model. The gap between a standard instruct model and a reasoning model is increasingly a gap in post-training compute, not pretraining scale. This means post-training now spans a wide spectrum from fast, cheap, highly iterable fine-tuning runs to massive RL campaigns that rival pretraining in both cost and engineering complexity. Why post-training matters So why can’t we just stick with pretraining? It comes down to three main pieces: usability, safety, and capability. Usability A pretrained model is like if someone gave you a large download of Wikipedia in a single PDF. It’s a ton of knowledge that you can sift through, but there’s no way to easily understand what is going on in the data. Post-training gives the model the ability to integrate this information for you and respond to your request naturally. This extends to having longer multiturn conversations and following instructions. Without it, every user would need to be a prompt engineer. With it, anyone who can type a sentence can use the model. Safety A lot of data in pretraining can be toxic, biased, misleading, or outright dangerous. Or it might not be dangerous on its own, but when a model can integrate knowledge from different fields, it can create something novel that is dangerous. The model has no inherent sense of what content is good or bad. It will follow any request, based on its pretraining data. To prevent that, you can add safety guardrails to the model in post-training, to refuse harmful requests like asking the model to build a bioweapon and avoid accidentally generating toxic content such as inappropriate sexual content (even if it wasn’t in the user’s request). This is also the place to teach the model to express uncertainty, when it doesn’t know something, whether that’s “I don’t know” or “that’s beyond my knowledge cutoff” or “as a large language model, I’m limited in my knowledge so please consult a healthcare professional.” Ma [truncated for AI cost control]