Kimi K3 on vLLM: Up to 370 Tokens/sec

We're thrilled to announce efficient day-0 vLLM support for Kimi K3, one of the most powerful open-weight models ever released.

Last week, we previewed the production-scale integration work for Kimi K3; today, Moonshot AI's weights are public and the support is live.

Kimi K3 day-0 support on vLLM

Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model (16 of 896 experts active per token) built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), with a 1M-token context window and native vision. For us, the most exciting challenge Kimi K3 brings is making KDA, MXFP4 MoE, KV cache management, prefill/decode disaggregation, speculative decoding, and long-context deployment recipes work together in a runnable serving engine.

The preview post explained the kernel and cache architecture, in particular the challenge of prefix caching that works on recurrent state. This release post is the practical guide: how vLLM adapts to Kimi K3's architecture, the kernel work behind the numbers, and what is ready on day 0.

Quick start

See the linked recipes for the exact Docker command.

vllm serve moonshotai/Kimi-K3 \ --tensor-parallel-size 8 \ --trust-remote-code \ --load-format fastsafetensors \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser kimi_k3 \ --reasoning-parser kimi_k3

The easiest way to run Kimi K3 is to use 8 NVIDIA B300 GPUs or 8 AMD MI355X GPUs with the above command.

Inferact has also trained and open-sourced a DSpark speculator for Kimi K3. Enable it by adding the following option to the serve command:

--speculative-config '{"model":"Inferact/Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'

For more details, including Docker images for various platforms and deployment strategies, refer to the detailed recipes. Because of complicated dependencies, only Docker images are usable now. The Docker images depend on several pre-release dependencies, including FlashInfer.

TL;DR

A 2.8-trillion-parameter multimodal MoE: Kimi K3 activates 16 of 896 experts per token, supports a context window of up to 1M tokens, and is built on Kimi Delta Attention, Attention Residuals, LatentMoE, and native MXFP4 (4-bit) weights.

Up to 370 tok/s per user: vLLM serves Kimi K3 at 118 tok/s without speculative decoding and 370 tok/s (a 3.14× improvement) with DSpark on 16 NVIDIA GB300 NVL72 GPUs, powered by extensive optimizations for Kimi K3's architecture.

Broad production feature support: vLLM supports speculative decoding, prefill/decode disaggregation, agentic KV caching with Mooncake, tool calling, reasoning output, and structured output, with NVIDIA (Hopper and Blackwell) and AMD (MI355X) support at launch.

Open-source DSpark support: vLLM supports the state-of-the-art block-diffusion speculative decoding algorithm for Kimi K3, trained with vLLM and TorchSpec and open-sourced by Inferact.

Hybrid prefix caching: serving Kimi K3's recurrent and full-attention design required a redesign of hybrid prefix caching over recurrent KDA state. This change now benefits every hybrid linear model.

Kimi K3's architecture, and how vLLM serves it

Kimi K3 architecture innovations

Kimi K3 architecture innovations, from the original release blog post.

Kimi K3's architecture departs from a standard Transformer in a few ways, and each one changes what a serving engine has to do. The preview post covers the internals in depth; here we recap what's new and focus on how vLLM adapts to serve it.

Kimi Delta Attention: a hybrid recurrent + full-attention stack

What's new: Most of Kimi K3's layers are KDA, a linear-attention mechanism that keeps a fixed-size recurrent state instead of a growing KV cache, interleaved with periodic full-attention layers that preserve exact global recall. That is what makes a 1M-token context affordable.

How vLLM serves it: A single hybrid KV-cache manager holds two kinds of memory side by side under one scheduler: paged KV blocks for the full-attention layers, and compact recurrent-state blocks for the KDA layers. A dedicated KDA attention backend runs FlashKDA for prefill and a fused CUDA kernel (or the Flash-Linear-Attention/Triton path when running speculative decoding) for decode.

The hardest part is prefix caching across Kimi K3's hybrid cache: full-attention layers store per-token KV, while KDA layers update recurrent and convolution state at every token but cannot afford to retain a snapshot at every possible prefix boundary. vLLM decouples the large physical KDA state blocks from fine-grained prefix matching, registering state snapshots within those blocks and copying them before extension so long shared prompts can reuse both KDA state and paged KV. This hybrid-cache machinery is new in vLLM core and now benefits every hybrid model similar to Kimi K3.

Kimi K3's hybrid KDA and full-attention cache

Kimi K3 interleaves Kimi Delta Attention layers with periodic full-attention layers; vLLM's hybrid cache manages recurrent state and paged KV together.

Attention Residuals: learned mixing of residual contributions across depth

What's new: For each token, Block AttnRes replaces ordinary residual accumulation with depth-wise attention: every Transformer sublayer uses a learned pseudo-query to weight RMS-normalized residual states from preceding layer blocks, then receives the corresponding weighted combination as its input.

How vLLM serves it: vLLM uses optimized Triton and CUDA kernels to compute the depth-wise attention logits, softmax, and hidden-state aggregation in a single fused operation. Residual updates and output RMSNorm are folded into the same kernel where supported, reducing intermediate memory traffic and kernel-launch overhead in both prefill and decode.

Stable LatentMoE: quantile-balanced latent-space experts at 16-of-896 sparsity

What's new. LatentMoE, introduced by NVIDIA, projects dispatched token activations into a narrower latent dimension for routed-expert computation, then projects the combined expert output back to the model width—reducing expert-weight bandwidth and all-to-all traffic so more experts can be used at similar inference cost. Kimi K3's Stable LatentMoE scales this design to 896 experts with 16 active per token and uses Quantile Balancing to derive expert allocation from router-score quantiles instead of heuristic balancing updates.

How vLLM serves it: Experts are sharded with expert parallelism. vLLM offers two MoE backends tuned for different topologies: TRT-LLM-Gen for tensor-parallel (TP > 1) and MegaMoE for disaggregated/expert-parallel (DEP). It also supports optional Expert-Parallel Load Balancing (EPLB) to ensure each rank has a similar amount of compute. The weights execute natively in MXFP4 on the MoE path.

Chat template: a render program, not a Jinja template

What's new: Kimi K3's chat template must encode system, user, and assistant messages, multimodal content, tool definitions, and tool results using exact control tokens. Instead of the common approach of a Jinja chat template that renders the request as text before tokenization, Kimi K3 uses a Python program to build the prompt token sequence directly. Its output likewise contains distinct regions for reasoning, answer text, and tool calls that must be parsed into an API response.

How vLLM serves it: vLLM implements both the input renderer and streaming output parser in its Python and Rust frontends, preserving control-token boundaries while treating user-supplied and tool-supplied text as ordinary content. For tool calls and structured outputs, vLLM integrates Kimi K3's format with XGrammar so structured regions are constrained during decoding and returned as separate reasoning, content, and tool-call fields.

Built for production

Serving a 2.8T hybrid MoE well means being fast for each user, efficient for many concurrent sessions, and scalable for agents. vLLM ensures Kimi K3 is ready on all three.

Ultra-low latency: speculative decoding with DSpark

To reach ultra-low latency on a 2.8T-parameter model like Kimi K3 without accuracy loss, speculative decoding is the natural choice. That is why vLLM supports DSpark, a state-of-the-art speculative decoding algorithm, from day 0—and why we trained and released a DSpark speculator for Kimi K3. The draft model is trained with vLLM using TorchSpec to achieve full numerical parity between speculator inference and training.

DSpark uses a block-diffusion backbone to generate multiple speculative tokens in one parallel pass based on Kimi K3's rich intermediate states, so drafting cost stays flat as the block deepens. A low-rank Markov head supplies the intra-block dependency, and a confidence head predicts the likelihood for each draft to be accepted. We made the draft MLA-native, mirroring Kimi K3's own attention, so draft and target share a similar KV layout to be maximally compatible with advanced KV management and P/D-disaggregated setups.

Kimi K3 DSpark positional acceptance rates

Kimi K3 DSpark positional acceptance rates across various datasets.

With DSpark, we achieve a 3.14× speedup on a single-user request, from 118 tok/s to 370 tok/s, measured using SPEED Bench. We also benchmarked the acceptance rate and speedups on different tasks, with the results shown above. For coding and other low-entropy tasks, we achieve around 4.73 accepted tokens per step. For high-entropy tasks such as creative writing, we achieve around 2.61 accepted tokens per step.

Confidence-based scheduling with DSpark is an ongoing effort in vLLM. Once enabled, it uses the confidence head included in the DSpark model to predict how likely each drafted token is to be accepted, prioritizing strong proposals and pruning weak ones so verification is not spent on tokens that will not survive.

Both the draft model and the inference support are open source as of this release. See the deployment guide below to enable it.

DSpark draft-and-verify flow for Kimi K3

A lightweight DSpark draft proposes candidate tokens that Kimi K3 verifies in a single parallel pass, accelerating single-stream decode.

Sequence parallelism for TEP prefill

Sequence parallelism for TEP prefill

Sequence parallelism shards token ownership across ranks; the attention residual is applied per shard, and one all-gather rebuilds the full batch before the next layer's QKV projection.

For the prefill phase, we combine attention tensor parallelism with MoE expert parallelism (TEP). Compared with pure TP, TEP reduces communication overhead and keeps whole experts on each rank, yielding more efficient expert GEMM shapes.

However, the naive TEP implementation performs two all-reduce operations per layer—one after the attention output projection and one after the MoE—so every rank materializes the full batch and redundantly applies the attention residual to all of it. To address this, we implement sequence parallelism: the all-reduce after o_proj is replaced with a reduce-scatter so that each rank owns a shard of the tokens, the attention residual is applied per shard, the MoE's all-to-all performs dispatch and combine, and a single all-gather restores the full batch before the next layer's QKV projection.

This design provides two key advantages:

Reduced communication overhead: Reduce-scatter + all-to-all dispatch + all-to-all combine + all-gather are theoretically cheaper than two all-reduces. In practice, however, NCCL's reduce-scatter and all-gather are not optimized for prefill's message sizes and yield no speedup. We therefore implement custom reduce-scatter and all-gather kernels that are 1.7×–4.5× faster than NCCL, especially at small-to-medium message sizes.

Sharded attention residual: The attention residual stays sharded across ranks throughout the layer, so each rank computes and maintains only its shard of the tokens rather than the entire batch. This matters

[truncated for AI cost control]