待翻译：Efficient Decode Context Parallelism with vLLM for Long Context Workloads

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

1. Introduction Long-context inference is becoming essential for agentic AI, where assistants may need to reason over large code repositories and long chat histories. Agent-trace benchmarks now run from 64K all the way to 1M tokens and their KV caches are correspondingly large. Under a baseline tensor-parallel (TP) setup, this KV cache is partitioned by attention head, which puts a hard floor on how much it can shrink. Modern models use one of two attention schemes, and both hit this floor. Grouped-query attention (GQA) models store a small number of KV heads, and TP can only split the KV cache down to one head per GPU; once TP exceeds the number of KV heads, the cache starts duplicating across GPUs. Multi-head latent attention (MLA) models make this even worse: MLA compresses the Key/Value into a single low-rank latent vector shared across all query heads, so it effectively has only one KV head. Under normal TP there is nothing to split by head, meaning the latent KV cache is replicated in full across every TP rank. In both cases the duplicated KV cache eats into GPU memory, leaving very little room to serve additional requests. This caps the number of concurrent requests the system can handle, driving down throughput and pushing up cost per token. Decode Context Parallelism addresses this by splitting KV cache across the GPUs so each GPU stores and reads only part of the KV cache. This frees up GPU memory, allowing each GPU to take on more requests and thus run at a larger batch size. On systems with high-bandwidth GPU-to-GPU interconnects, this helps preserve interactive responsiveness while serving many long-context agents at once. vLLM has supported DCP for almost a year, but we are writing this blog now to highlight the feature, along with the recent improvements and advancements we have made to it, because the rise of long-context agentic use cases has made its benefits more relevant than ever. 2. Performance Results To quantify the benefit of Decode Context Parallelism, we compared a baseline tensor-parallel deployment against DCP on an identical set of GPUs, holding the model, hardware, and workload fixed and varying only how the KV cache is sharded during decode. 2.1 Dataset The dataset is a publicly available agentic long-context trace in Mooncake trace format, published here. See this section for more details on the dataset. It ships as JSONL where each line is a single request with input_length, output_length, and hash_ids fields, so it can be replayed directly with any Mooncake-compatible harness (e.g. aiperf --custom-dataset-type mooncake_trace). The hash_ids field encodes shared prefix blocks, making it well-suited for benchmarking KV-cache reuse and prefix-caching behavior. It's an agentic multi-turn workload of long inputs paired with short generations, chosen to reflect realistic long-horizon agent behavior. Inputs are centered around a median of ~67k tokens and paired with short ~400-token outputs, but the input distribution is bimodal rather than uniformly huge: roughly half the requests sit at 64k+ (≈53%, with a heavy tail reaching ~1M tokens) and half are short-to-mid (≈47% under 64k, ~18% under 8k). About 8% of requests exceed 128k and ~3–4% exceed 256k. 2.2 Benefits of Decode Context Parallelism We ran an experiment on a single 8×B200 node serving Kimi K2.6 in NVFP4 with vLLM, sweeping request concurrency from 16 to 512 (see table below). DCP sustains far higher concurrency and delivers markedly higher throughput per GPU across the entire throughput–interactivity Pareto frontier. The difference comes down to where the KV cache lives. Baseline TP replicates the KV cache on every GPU, so peak memory fills quickly. It reaches 100% at a concurrency of 64 and hits a wall, and throughput plateaus near 1,863 tok/s/GPU because no additional requests can fit. On the other hand, DCP shards the KV cache along the sequence dimension, so each GPU stores only 1/N of every request's KV. This allows space on the GPU to support more incoming requests. As a result, even at high concurrencies DCP keeps scaling where TP hits a wall. DCP reaches 6,091 tok/s/GPU at c512 while still sitting at just 82% KV usage. The core value of DCP is that it sustains far higher concurrency, even on long-context runs, precisely the regime where replicated-KV TP runs out of memory first. 2.3 Comparison by Sequence Length We also plotted performance against full sequence length (input + output). The figure shows a single throughput–interactivity Pareto frontier with requests grouped into five length bands (= decode_context_parallel_size tensor_parallel_size % decode_context_parallel_size == 0 vllm serve deepseek-ai/DeepSeek-R1 \ --tensor-parallel-size 8 \ --decode-context-parallel-size 8 5.4 GQA Backend Example models: Qwen3-235B, and other Grouped-Query-Attention models (Llama-family, etc.). Why it's different. GQA stores num_key_value_heads KV heads, and TP splits the KV cache by those heads first. That works cleanly only up to num_key_value_heads; once tensor_parallel_size exceeds it, the KV cache begins duplicating, with tp // num_key_value_heads identical copies across ranks. What they do. DCP takes those would-be-duplicate copies and fills them with different sequence chunks instead, while the shared KV heads are broadcast across their query heads (the "tensor broadcast for GQA" step). So the sequence-split degree is capped by the duplication factor tp // num_key_value_heads: (tensor_parallel_size // num_key_value_heads) >= decode_context_parallel_size (tensor_parallel_size // num_key_value_heads) % decode_context_parallel_size == 0 # Qwen3-235B has num_key_value_heads = 4; tp=8 gives 8//4 = 2 redundant copies, # so dcp can be up to 2. vllm serve Qwen/Qwen3-235B-A22B \ --tensor-parallel-size 8 \ --decode-context-parallel-size 2 6. Future Work Looking ahead, we plan to extend DCP along several main directions. We will add support for finer-grained parallelism sizes for both TP and DCP, giving users more precise control over their parallelism layout and reclaiming efficiency lost to over-provisioned sharding. We are also developing better DCP all-to-all (A2A) communication kernels for both multinode and single-node settings, reducing exposed communication and improving overlap with compute as context length and device count grow. We are working on better support for MTP and speculative decoding, so that DCP can deliver its efficiency gains without sacrificing the latency benefits of speculative methods, as well as hardening prefill/decode (P/D) disaggregation support to make DCP robust in disaggregated serving deployments. Finally, we aim to broaden DCP's reach by extending support to a wider variety of backends and integrating it with hybrid models and Dynamic Chunked Pipeline Parallelism, so a much wider range of workloads can benefit from context-parallel efficiency gains. The community is also expanding DCP to additional models such as GLM-5.2 and Kimi K3, and there is a longer roadmap for Prefill Context Parallelism (PCP). We are working on DCP performance benchmarking for the Kimi K3 model and plan to share those results as that work matures. For deployment guidance and historical notes on DCP, see the vLLM Decode Context Parallel docs. 7. Conclusion Decode Context Parallelism represents a fundamental rethinking of how GPUs are organized for long-context inference. Rather than forcing GPUs to duplicate KV cache or sit underutilized, DCP puts every GPU to work: sharding the sequence during attention, then immediately reconfiguring those same GPUs to amortize FFN weight loading across the full pool. The result is a system that scales gracefully with context length rather than degrading under it. With native support in vLLM, Decode Context Parallelism is ready to power the next generation of long-context agentic applications, from document reasoning to multi-session agentic pipelines, at the throughput and latency that production demands. It joins a broader industry move toward Decode Context Parallelism, a direction NVIDIA has also pursued with Helix Parallelism in TensorRT-LLM. We are also working on DCP performance benchmarking for the Kimi K3 model and plan to share those results as that work matures. About Us Special thanks to the NVIDIA team Anahita Bhiwandiwalla, Xin Li, Pavani Majety, Nidhi Bhatia, Roman Ageev, Pen Chung Li, and Chris Hoge for their reviews, benchmarking support, and engineering input throughout this study. We also thank Moonshot AI for the initial Decode Context Parallel work upstreamed in vLLM #23734, and Lucas Wilkinson for substantial follow-up contributions that helped harden and extend DCP. We also thank the broader vLLM community, whose open-source engine and continued collaboration made this benchmarking effort possible. For more on DCP deployment and related history, see the vLLM Decode Context Parallel docs. The DCP results in this post were measured on NVIDIA B200 GPUs with Kimi K2.6 in NVFP4, and the recipes can be reproduced with current vLLM releases that support --decode-context-parallel-size. We are also working on DCP performance benchmarking for the Kimi K3 model and plan to share those results as that work matures.