AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。
LLM Batching Explained: Static vs Dynamic vs Continuous Batching - Analytics Vidhya India's Most Futuristic AI Conference Is Back – Bigger, Sharper, Bolder d : h : m : s Career GenAI Prompt Engg ChatGPT LLM Langchain RAG AI Agents Machine Learning Deep Learning GenAI Tools LLMOps Python NLP SQL AIML Projects Reading list How to Become a Data Analyst in 2025: A Complete RoadMap A Comprehensive Learning Path to Tableau in 2025 A Comprehensive NLP Learning Path 2025 Learning Path to Become a Data Scientist in 2025 Step-by-Step Roadmap to Become a Data Engineer in 2025 A Comprehensive MLOps Learning Path: 2025 Edition Roadmap to Become an AI Engineer in 2025 A Comprehensive Learning Path to Master Computer Vision in 2025 Best Roadmap to Learn Generative AI in 2025 GenAI Roadmap for Enterprises Large Language Models Demystified: A Beginner’s Roadmap Learning Path to Become a Prompt Engineering Specialist LLM Batching Explained: Static vs Dynamic vs Continuous Batching Soumil Jain Last Updated : 12 Oct, 2026 9 min read Imagine four people booking a ride to the same destination. If each person takes a separate taxi, four vehicles make the same trip. But if they travel together in a bus, everyone reaches the destination in a single trip. The bus carries more people at once and makes better use of the resources. Batching in AI works in a similar way. Instead of processing one request at a time, a server groups multiple requests together so the GPU can process them more efficiently. This helps serve more users while making better use of expensive computing resources. But how those requests are grouped and scheduled also affects how quickly users get responses and how much it costs to serve them. There are three common approaches to batching: static batching, dynamic batching, and continuous batching. Let’s understand what batching means technically and how these approaches differ. Table of contents What is Batching? Why LLMs Need Batching? Static Batching Dynamic Batching Continuous Batching Static vs Dynamic vs Continuous Batching Conclusion What is Batching? Batching is the process of running multiple inputs through a model in a single forward pass instead of processing each input individually. In an LLM, each input is a sequence of tokens, so the model processes a batch of sequences together. Conceptually, instead of an input tensor shaped [1, seq_len], the model processes a batch shaped [batch_size, seq_len]. This allows the GPU to perform computations across multiple requests in parallel, improving hardware utilization and throughput. However, LLMs generate responses token by token, and different requests can take different amounts of time to finish. How the server groups requests, handles incoming traffic, and frees up resources as responses finish determines which batching approach works best. Why LLMs Need Batching? The answer is simple, during generation, the GPU is limited by how fast it can read memory, not how fast it can do the math. Batching lets one memory read serve many requests. LLM inference has two phases: Prefill: The model reads your whole prompt in one forward pass. All prompt tokens are processed in parallel, so this phase keeps the GPU’s compute units busy. Decode: The model generates one token at a time. Each new token needs a full forward pass, which means reading every model weight from GPU memory again. Decode is where the waste happens. Some rough numbers for a 7B model in FP16 on an H100: The weights take about 14 GB. H100 memory bandwidth is about 3.35 TB/s, so reading 14 GB takes roughly 4 ms. That caps a single request at around 240 tokens per second. Each token needs about 14 GFLOPs (2 × parameters). Doing that in 4 ms is around 3.5 TFLOPS, while the H100 offers close to 1,000 TFLOPS of dense BF16. So at batch size 1, the GPU’s compute sits more than 99% idle. It spends almost all its time loading the weights. If you run 32 sequences in the same steps, the weights are loaded once and you get 32 tokens instead of 1, nearly the same step time. That’s the whole case of batching higher throughput and lower cost per token, without buying more GPUs. There is a limit. Every sequence in the batch keeps its own KV cache in GPU memory, and that cache grows with every token. Batch size ends up bounded by memory, which becomes important when we get to continuous batching. Also Read: The Four Caches in LLM Serving Static Batching Static batching groups a fixed set of requests, runs them together, and returns results only when every request in the group is finished. Nothing new can join until the whole batch is done. The model.generate() call from earlier is static batching. It’s the default behavior of most hand-written inference loops. The problem is that LLM outputs vary wildly in length. Say a batch of 4 requests ends up generating 20, 50, 100 and 500 tokens. Then the batch runs for 500 decode steps, because it waits for the longest one. The 20-token request is done at step 20, but its slot sits idle (or computes padding) for 480 more steps. Useful work is 20 + 50 + 100 + 500 = 670 token-steps out of 4 × 500 = 2,000 possible. That’s about 33% utilization.A request that arrives at step 1 has to wait 499 steps before it can even start. This is called head-of-line blocking: one long request holds up everyone behind it. To compare the three strategies fairly, let’s build a small simulator. It skips the model and only tracks decode steps, which is enough to see how scheduling affects GPU usage and latency. A new request arrives every 10 steps, and the GPU can run 4 sequences at once. Code: from collections import deque # Each request: (id, arrival_step, output_tokens) # A new request arrives every 10 decode steps, with very different output lengths. lengths = [120, 30, 60, 300, 40, 90, 25, 150, 50, 70, 35, 80] REQUESTS = [(f"R{i}", i * 10, n) for i, n in enumerate(lengths)] SLOTS = 4 # max sequences the GPU runs at once def summarize(name, total_steps, useful, finish): latency = [finish[rid] - arr for rid, arr, _ in REQUESTS] util = useful / (total_steps * SLOTS) print(f"{name: tokens left step, useful, finish = 0, 0, {} while waiting or running: # every step: fill free slots with requests that have arrived while waiting and len(running) < SLOTS and waiting[0][1] <= step: rid, _, n = waiting.popleft() running[rid] = n useful += len(running) step += 1 for rid in list(running): running[rid] -= 1 if running[rid] == 0: # done: return now, free the slot finish[rid] = step del running[rid] return step, useful, finish if name == "main": summarize("Static", *static_batching(REQUESTS)) summarize("Dynamic", *dynamic_batching(REQUESTS)) summarize("Continuous", *continuous_batching(REQUESTS)) The difference is that the scheduling loop runs inside the decode loop. Each request’s finish time is its own, not the batch’s. What makes it work in practice: The idea is simple, but real systems need a few extra pieces: Mixing prefill and decode: New requests need prefill while existing ones generate tokens. Orca uses selective batching to handle both efficiently. KV cache management: vLLM’s PagedAttention stores KV caches in fixed-size blocks, reducing memory fragmentation and allowing more sequences to fit in a batch. Chunked prefill: Systems like Sarathi-Serve split long prompts into smaller chunks and interleave them with decode steps to reduce delays for running requests. Static vs Dynamic vs Continuous Batching Running the full simulator on the same 12 requests gives this: Static steps=560 slot_util=47% avg_latency=402 first_two=[330, 320] Dynamic steps=665 slot_util=39% avg_latency=418 first_two=[135, 125] Continuous steps=330 slot_util=80% avg_latency=118 first_two=[120, 30] A few things stand out: Continuous batching finishes the same work in 41% fewer steps (330 vs 560) and cuts average latency from 402 steps to 118. The GPU slots are busy 80% of the time instead of 47%. Dynamic batching helps the first requests get results about 2.5x sooner than static, since it doesn’t wait for a full batch. But it does worse overall, because it launched half-empty batches that then had to run to completion. That’s the trade-off you get when a request-level batcher meets variable-length generation. R1 asked for only 30 tokens. With static batching it waited 320 steps for its answer. With continuous batching it waited exactly 30. The simulator ignores prefill cost and memory limits, so real numbers will differ, but the pattern holds on real serving engines. Static Dynamic Continuous When a batch starts Once a fixed group is ready When full or the timeout hits Every decode step New requests join mid-run? No No Yes Finished sequences Wait for the whole batch Wait for the whole batch Returned at once, slot reused Scheduling level Per batch Per batch Per iteration (token) Best fit Offline jobs, similar output lengths Fixed-cost models (BERT, embeddings, vision) LLM serving with variable outputs Where you’ll see it Plain model.generate() loops Triton dynamic batcher vLLM, TGI, TensorRT-LLM, SGLang If you remember one line: static batching waits for everyone, dynamic batching waits less to start but still waits for everyone to finish, and continuous batching doesn’t wait for anyone. That’s why every serious LLM serving stack today is built on continuous batching, and why swapping a hand-rolled generate() server for vLLM is often the cheapest speedup you’ll ever get. Conclusion Batching exists because LLM decode is memory-bound. Reading the weights once and using them for many sequences is the single biggest lever you have on cost per token. The three strategies are really a story of moving the scheduling decision to a finer level. Static batching decides once per group. Dynamic batching decides once per batch but stops waiting for a full group. Continuous batching decides on every token, so short requests never get stuck behind long ones. For most teams, the practical takeaway is simple: don’t build your own batching loop for LLM serving. Use an engine like vLLM, TGI or TensorRT-LLM, then spend your time tuning the knobs that matter, such as maximum concurrent sequences, tokens per step, and how much GPU memory goes to the KV cache. Soumil Jain I am a Data Science Trainee at Analytics Vidhya, passionately working on the development of advanced AI solutions such as Generative AI applications, Large Language Models, and cutting-edge AI tools that push the boundaries of technology. My role also involves creating engaging educational content for Analytics Vidhya’s YouTube channels, developing comprehensive courses that cover the full spectrum of machine learning to generative AI, and authoring technical blogs that connect foundational concepts with the latest innovations in AI. Through this, I aim to contribute to building intelligent systems and share knowledge that inspires and empowers the AI community. IntermediateLLMs Login to continue reading and enjoy expert-curated content. Free Courses 0 The Forward Deployed Engineer: Taking AI From Pilot to Production Learn how Forward Deployed Engineers take AI from pilot to production. 0 FastAPI for AI Engineers: The Complete Guide to Building Scalable AI APIs Build scalable AI APIs using FastAPI and GenAI workflows. 0 Mastering Claude Cowork Master Claude Cowork, build AI agents, and automate workflows using Claude. 0 Claude Code: The Coding Assistant Learn to create powerful apps and agents using Claude Code's AI assistant. 0 Stop Doing It Manually: Tasks You Should Hand to Cowork Automate real corporate tasks using Cowork AI. Recommended Articles GPT-4 vs. Llama 3.1 – Which Model is Better? Llama-3.1-Storm-8B: The 8B LLM Powerhouse Surpa... A Comprehensive Guide to Building Agentic RAG S... Top 10 Machine Learning Algorithms in 2026 45 Questions to Test a Data Scientist on Basics. [truncated for AI cost control]