Quantization and Pruning Methods to Make Your LLM Leaner

--> Quantization and Pruning Methods to Make Your LLM Leaner - KDnuggets --> Join Newsletter A team fine-tunes a model for three weeks, gets the evaluation numbers they wanted, and then tries to actually serve it. The checkpoint alone is 140GB. That single number rules out almost every GPU a normal company has sitting in a rack, forces a rewrite of the deployment plan, and turns what should have been a launch week into a scramble for four A100s nobody budgeted for. That moment is more common than it should be, and it's almost always avoidable. The model didn't need to ship at full precision with every parameter intact. It needed to ship as the leanest version of itself that still does the job, and the two techniques that get you there — quantization and pruning — are neither exotic nor new. They're just underused by teams who assume "make it smaller" means "make it worse." This article walks through what each technique actually does, why skipping them costs real money and real latency, and then gets hands-on with five specific methods people are running in production right now, each with working code you can adapt today. What Quantization and Pruning Actually Are These two get lumped together constantly, and it's worth separating them cleanly before going any further, because they solve different problems in different ways. Quantization lowers the precision of the numbers a model is made of. A weight stored as a 16-bit floating-point number, something like 0.0023847, gets rounded and re-represented using fewer bits — an 8-bit integer or a 4-bit integer. The number of parameters in the model doesn't change at all. Every weight that existed before still exists. It just takes up less space and computes faster, the same way a high-resolution photo saved at a lower bit depth still shows every object in the frame, just with less precision in the shading. Pruning removes weights, or entire structures, outright. A connection between two neurons, an attention head, sometimes a full layer, gets deleted because the model turns out not to need it. The parameter count itself goes down. This is closer to editing a long document by actually cutting sentences that weren't adding anything, rather than just writing everything in smaller font. Both techniques shrink a model. They just shrink it along different axes, and as you'll see later in this article, they stack cleanly on top of each other rather than competing for the same job. Why This Matters Right Now The scale problem underneath all of this is easy to understate until you see the actual numbers. A 70 billion parameter model stored in FP16 needs around 140GB of VRAM just to load, which in practice means four A100 GPUs before a single request gets served, according to Pristren's breakdown of LLM compression costs. That's roughly \$80,000 to \$100,000 of hardware sitting idle before the model does anything useful. Quantization changes that math directly. Compress the same 70B model to 4-bit using AWQ or GPTQ, and it drops to somewhere around 35 to 40GB — small enough to fit on a single high-end workstation card instead of a small cluster, as Fungies' 2026 quantization guide lays out. Far from a marginal optimization, this is actually the difference between a model that needs a data center and one that runs on hardware a single engineer can have under their desk. This isn't a niche concern restricted to hobbyists trying to run models locally, either. It's shaping how the biggest labs ship models in 2026. Google's Gemma 3 took its 27B model from 54GB down to roughly 14GB at 4-bit while cutting the quality loss against plain post-training quantization roughly in half, and its successor, Gemma 4, went further still, shipping quantization-aware checkpoints that get the smallest 2B variant down to about 1GB — small enough to run entirely on a phone — according to TensorFoundry's field guide to 2026 quantization. Apple's on-device models on current iPhones use the same trick, squeezing weights down to 2 bits through quantization-aware training rather than guessing at scales after the fact. The practical gains: fewer GPUs to buy or rent, lower latency per request since less data has to move through memory, and the ability to put real capability on hardware that was never going to hold the full-size model in the first place. What Happens If You Skip This, or Do It Badly The flip side is worth covering honestly, because both directions of failure show up constantly in practice. Skip compression entirely, and the failure is usually simple and expensive: a model too large to deploy on the hardware you actually have, an inference bill that makes the product commercially unviable, or latency high enough to break any use case that needs a fast response — a live chat interface, a voice assistant, an autocomplete tool. None of that is hypothetical. It's the default outcome for any team that trains a large model and assumes serving it will be someone else's problem to figure out later. The opposite failure is quieter and more dangerous, because it doesn't announce itself the way an out-of-memory error does. Quantize too aggressively, without a proper calibration dataset, or ignore the small number of outlier weights that carry a disproportionate amount of a model's actual capability, and accuracy degrades in ways that don't always show up in a quick smoke test. Red Hat's own study covering more than 500,000 evaluations of quantized models found that quality loss varies significantly by model, task, and method — some models tolerate aggressive compression fine, others fall apart fast, and the only way to know which you're dealing with is to actually benchmark the compressed version on tasks that resemble what it'll be used for, not just check that it still produces grammatical sentences. Prune carelessly, and the same pattern shows up: research on plain magnitude pruning, the simplest possible approach, found it fails dramatically on large language models (LLMs) even at fairly modest sparsity levels, as the team behind the Wanda pruning method documented directly. LLMs turn out to be substantially harder to prune safely than the smaller networks that magnitude pruning was originally designed for. The five methods in this article exist specifically to sit in the middle of those two failure modes: real, meaningful compression, done carefully enough that it doesn't quietly wreck the model you spent weeks building. The Five Methods, at a Glance Before going deep on each one, here's the map. Three are quantization methods, two are pruning methods, and they differ meaningfully in how much setup they need and what they're actually optimized for. Method Category Typical size reduction Retraining needed Best fit bitsandbytes (NF4) Quantization ~4x No (supports optional fine-tuning via QLoRA) Fast setup, and the only option here that also enables fine-tuning GPTQ Quantization ~4x No, calibration only Mature GPU serving, wide pre-quantized model availability AWQ Quantization ~4x No, calibration only Production GPU serving, best quality-to-speed ratio on modern kernels SparseGPT Pruning ~2x (at 50% sparsity) No, one-shot with weight update Large models, structured 2:4 sparsity for real hardware speedups Wanda Pruning ~2x (at 50% sparsity) No, single forward pass Very large models where pruning speed itself matters Method 1: bitsandbytes (NF4 4-Bit Quantization) This is the method most teams should reach for first, and it's a little undersold in a lot of guides precisely because it's simple enough to use in a single function call. It's built around a data type called NF4 — NormalFloat4 — designed specifically around the fact that neural network weights tend to follow a roughly normal distribution rather than being spread evenly across the number line, so the available 4-bit values are placed where the actual weights cluster instead of being spaced out uniformly. It's also the one method on this list that supports QLoRA, meaning you can load a model in 4-bit and still fine-tune it by training small low-rank adapter weights on top, without ever touching the frozen 4-bit base weights directly. If fine-tuning is anywhere in your plan, this is the natural starting point. from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig import torch model_id = "meta-llama/Llama-3.1-8B-Instruct" # Configure 4-bit NF4 quantization with double quantization enabled bnb_config = BitsAndBytesConfig( load_in_4bit=True, # load weights in 4-bit instead of 16-bit bnb_4bit_quant_type="nf4", # NormalFloat4: a data type tuned for # the normal-ish distribution of NN weights bnb_4bit_compute_dtype=torch.bfloat16, # matmuls are upcast to bfloat16 at # compute time, weights stay stored at 4-bit bnb_4bit_use_double_quant=True, # quantizes the quantization constants # themselves, saving roughly another # 0.4 bits per parameter on top ) tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, quantization_config=bnb_config, device_map="auto", # spreads layers across available # GPU(s), offloading to CPU if needed ) inputs = tokenizer("Explain quantization in one sentence.", return_tensors="pt").to(model.device) output = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(output[0], skip_special_tokens=True)) Walking through what actually matters here: load_in_4bit=True is the switch that triggers the whole process, converting every linear layer's weights to 4-bit on load rather than requiring a separate offline quantization pass first, which is exactly why this is the fastest method to get running. bnb_4bit_quant_type="nf4" picks the distribution-aware format over plain 4-bit integers, which is what keeps quality close to the original model instead of just rounding blindly. bnb_4bit_compute_dtype=torch.bfloat16 matters because the weights sit in memory at 4-bit but get temporarily upcast to bfloat16 during the actual matrix multiplication, since GPUs don't have native 4-bit compute kernels for this yet, so this line controls that intermediate precision. And bnb_4bit_use_double_quant=True is a small but genuinely free win: it quantizes the scaling constants used to quantize the weights in the first place, squeezing out a bit more memory with no meaningful accuracy cost. Method 2: GPTQ (Calibrated Post-Training Quantization) GPTQ was one of the first 4-bit methods that actually held up well on large models, introduced in the original GPTQ paper from Frantar and colleagues in 2022. The mechanism is what separates it from naive rounding: it quantizes a model layer by layer, and within each layer, it uses second-order information — an approximation of the Hessian matrix — to figure out how rounding one weight affects the ideal values of the weights around it, then adjusts the remaining unquantized weights in that layer to compensate for the error just introduced. It's error correction built directly into the quantization process, rather than quantizing every weight independently and hoping the errors don't compound. That mechanism needs a calibration dataset, typically a few hundred samples of representative text, to estimate those Hessian statistics accurately. from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig import torch model_id = "meta-llama/Llama-3.1-8B-Instruct" tokenizer = AutoTokenizer.from_pretrained(model_id) # GPTQConfig drives both calibration and quantization in a single pass gptq_config = GPTQConfig( bits=4, # target bit-width per weight dataset="c4", # calibration text used to estimate the # Hessian-based error compensation tokenizer=tokenizer, group_size=128, # weights are quantized in groups of 128, # balancing accuracy against compression ratio desc_act=False, # skips activation-ord [truncated for AI cost control]