待翻译：Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

Running large language model (LLM) inference at scale typically forces a KV cache trade-off: you either pay for oversized GPU instances to accommodate a growing KV cache, or you accept slow time-to-first-token (TTFT) as identical prompts get recomputed on every request. For teams deploying a broad catalog of publicly available foundation models (FMs), such as Qwen, Llama, DeepSeek, and others, across per-business-line endpoints, Retrieval Augmented Generation (RAG) pipelines, or multi-turn dialogue applications, this trade-off translates directly into higher infrastructure cost and degraded user experience. The root cause is straightforward. During generation, vLLM stores the attention keys and values for every token it has already processed in a KV cache, so it doesn’t recompute them on each step. Prefix caching extends this by reusing that cache across requests that share the same leading tokens (like a common system prompt). On cost-efficient instances like ml.g6e.4xlarge (48 GB per GPU), once model weights and runtime allocations are accounted for, the memory left for prefix caching is limited, and it tightens further with larger models or higher concurrency. Cache hit rates drop on long prompts, identical system prompts get re-prefilled on every request, and horizontally scaled vLLM replicas each maintain isolated caches. Routing to a different replica is functionally a cold start. In this post, we build a tiered KV cache architecture on Amazon SageMaker HyperPod that extends the cache hierarchy beyond GPU and CPU memory into a shared, distributed NVMe pool. It builds on two HyperPod capabilities, Managed Tiered KV Cache and Intelligent Routing, and adds Curvine, a lightweight distributed cache filesystem, as the shared L2 tier (GPU to CPU to shared NVMe). With this setup, you can reuse KV cache across replicas at near-local-disk speeds. We walk through the end-to-end implementation, from enabling HyperPod Tiered Storage to deploying Curvine workers on node-local NVMe to patching the Inference Operator for filesystem-backed L2. On a test deployment, this achieved up to a 100 percent cross-Pod cache hit rate, up to a 2.7x TTFT improvement, and cross-node L2 read latency of about 56 ms for a approximately 1,900-token prompt. See the Benchmarking section for the full methodology and results. With this architecture, workloads that previously required P5 instances can run on lower-cost G6e instances, reducing per-endpoint cost. Actual savings depend on model size and traffic profile. Solution design The central idea is to extend the KV cache beyond what fits on a single Pod. Rather than accepting that each vLLM replica lives in isolation, which is its own GPU blocks, its own CPU spill area, no sharing, we build a three-tier hierarchy: L0 (GPU HBM), L1 (local CPU/host memory), and L2 (Curvine, a shared cross-node cache), and overlay it with cache-aware request routing. L0 – GPU prefix cache. This is vLLM’s native paged-attention layer, holding the hottest KV blocks at the lowest access latency, but its capacity is only whatever GPU memory is left after the model weights. On a 48 GB GPU, a 7B model in bf16 uses around 14 GB for weights, leaving over 30 GB for KV blocks, which is plenty of headroom, so L0 pressure is minimal. A 32B model uses around 64 GB of weights and doesn’t even fit on one 48 GB GPU. Even after sharding, far less memory remains for KV, so the cache fills quickly and evicts under concurrency. That shrinking headroom is exactly why extending the cache off-GPU matters as you scale up model size and traffic. L1 – CPU memory offload. When GPU blocks are evicted, LMCache catches them in host DRAM before they’re lost. This runs inside each inference Pod and is managed automatically by the SageMaker HyperPod Inference Operator when you set enableL1Cache: true in the InferenceEndpointConfig CRD. Think of it as a safety net. It’s fast, Pod-local, and sized by InstanceMemoryAllocationPercentage (we recommend starting at 20 percent). L2 – Shared distributed NVMe pool. This is where cross-replica reuse happens. Curvine, a lightweight distributed cache filesystem, pools the local NVMe drives that ship with G6e/P5 instances into a single namespace, which a FUSE client (a user-space driver that presents the pool as an ordinary mounted directory) mounts as a ReadWriteMany PVC (PersistentVolumeClaim) into every inference Pod. LMCache reads and writes through its fs:// connector, so the distributed pool looks like a local directory. Because every Pod mounts the same namespace, a KV block written by one replica is immediately readable by others. Curvine itself is straightforward to operate: a Primary Node (called the “Master” in Curvine’s documentation) handles metadata and journaling, persisted on Amazon Elastic Block Store (Amazon EBS) for durability, while Worker components run on each GPU node and store data on the node’s NVMe (typically mounted at /opt/dlami/nvme/curvine-data). If a Worker dies, the cache it held is recomputed, no data-loss concern, since these are reproducible KV blocks. Intelligent routing – getting requests to the right replica. A three-tier cache only delivers its full benefit if requests land on replicas that already hold relevant KV blocks. The HyperPod Inference Operator includes a built-in router that supports three strategies: Strategy Best for prefix-aware (default) Multi-turn dialogue, shared system prompts kv-aware Long document processing, extended sessions round-robin Stateless batch inference, load testing The router maintains a prefix tree (prefix-aware) or queries each worker’s cache state (kv-aware) to select the replica most likely to produce a cache hit. This happens transparently, no client-side changes are needed. How these pieces fit together. The Inference Operator is installed as an Amazon Elastic Kubernetes Service (Amazon EKS) add-on and manages the full lifecycle. It spins up vLLM Pods with LMCache sidecars, configures L1 and L2 backends, deploys the router, and exposes a single load-balanced endpoint. You declare the cache topology you want in the InferenceEndpointConfig CRD (enableL1Cache, enableL2Cache, l2CacheBackend, routingStrategy), and the Operator renders the correct environment variables, volume mounts, and routing rules. The one caveat today: the CRD’s l2CacheBackend field only accepts redis or tieredstorage natively. To point L2 at a Curvine FUSE mount, we patch the LMCACHE_REMOTE_URL environment variable in the vLLM container spec to fs://localhost:0/mnt/curvine/l2cache/. We walk through this patch in Stage 4 of the implementation. The net effect is a request arrives at the router, gets dispatched to the replica with the best prefix match, that replica checks GPU blocks (L0), then CPU (L1), then the shared NVMe pool (L2). Only on a complete miss does it re-prefill from scratch. For workloads with moderate-to-high prompt overlap (roughly over 40 percent shared leading tokens, for example a common system prompt or shared RAG context), skipping that re-prefill substantially reduces TTFT. Figure 1 shows the full data path. Each vLLM Pod stacks an L0 GPU prefix cache and an L1 CPU offload. Below them, all Pods share the L2 tier on a Curvine distributed filesystem pooled from node-local NVMe and mounted ReadWriteMany over FUSE, while the Curvine metadata node persists to Amazon EBS. The HyperPod Intelligent Router sits in front, directing each request to the replica most likely to already hold the relevant cache. Figure 1: Tiered KV cache architecture Curvine is a high-performance distributed cache file system that sits between applications and underlying storage such as Amazon Simple Storage Service (Amazon S3), HDFS, or NAS. Clients reach it through the CLI, SDK, FUSE, or CSI. Primary Nodes handle metadata, and Workers serve data with local disk cache for low-latency I/O. Figure 2 shows the Curvine architecture and its key components. Figure 2: Curvine architecture How Curvine works (cluster view): Clients send metadata RPC to Masters and data I/O to Workers. Masters coordinate Workers using heartbeats and place blocks for load balance and HA. Workers read/write local tiers and promote/demote data by heat. On miss or policy-driven persistence, Curvine loads from / dumps to UFS, so durability stays on the underlying store while Curvine accelerates access. Prerequisites Amazon SageMaker HyperPod Tiered Storage is a cluster-level capability that provisions a node-local cache tier for inference workloads. After Tiered Storage is active, SageMaker HyperPod deploys the ai-toolkit DaemonSet on every GPU node, reserves a configurable share of host memory (InstanceMemoryAllocationPercentage) for the L1 CPU offload, and exposes the local NVMe instance store under /opt/dlami/nvme so that Curvine Workers can pool it into a shared L2 namespace. The Inference Operator consumes these tiers automatically when enableL1Cache and enableL2Cache are set on the InferenceEndpointConfig CRD. This walkthrough assumes a SageMaker HyperPod cluster orchestrated by Amazon EKS. To create one, follow Orchestrating SageMaker HyperPod clusters with Amazon EKS in the SageMaker documentation, or with AWS CloudFormation, using the reference templates on the AWSome Distributed AI repository. Provision at least two GPU nodes. A single node can’t demonstrate cross-node reuse. Throughout this post we use the cluster name hyperpod-cluster-eks and the US West (Oregon) AWS Region (us-west-2) as examples, replace them with your own cluster name and Region to reproduce this solution in your account. Verify the following are in place: GPU capacity with local NVMe: A SageMaker HyperPod EKS cluster with at least one GPU instance group. G6e or P5 is recommended for their local NVMe, which Curvine pools into L2. CLI tooling: On your workstation: AWS Command Line Interface (AWS CLI) v2 (with permissions for sagemaker:UpdateCluster and eks:CreateAddon), kubectl configured against the cluster with aws eks update-kubeconfig, and Helm v3. AWS Identity and Access Management (IAM) for EBS attach: Grant the EBS CSI driver role sagemaker:AttachClusterNodeVolume, sagemaker:DetachClusterNodeVolume, and eks:Describe* so the Curvine metadata node can attach its EBS volume. Keep the Amazon Virtual Private Cloud (Amazon VPC) CNI and EBS CSI add-ons current. Model weights: this post pulls Qwen2-7B from HuggingFace, so no bucket is required. To stage weights yourself, use an Amazon S3 bucket with the SageMaker HyperPod execution role granted read access. TLS certificates are generated automatically. Tiered Storage is enabled in Stage 1. The Inference Operator, Amazon S3 and Amazon FSx CSI drivers, Metrics Server, and Cert Manager are installed in Stage 2 (or through console Quick Install), the EBS CSI driver and Curvine in Stage 3. Step-by-step implementation The following procedure uses this implementation as a worked example, organized into five stages. Cluster names and Regions shown are placeholders, substitute your own. Stage 1: Enable HyperPod Tiered Storage Tiered Storage is a cluster-level toggle. Once Tiered Storage is active, HyperPod automatically deploys the ai-toolkit DaemonSet to every node. # Enable on an existing cluster via update-cluster (recommended) aws sagemaker update-cluster \ --cluster-name hyperpod-cluster-eks \ --tiered-storage-config Mode=Enable,InstanceMemoryAllocationPercentage=20 \ --node-recovery Automatic API note. Calling update-cluster with --tiered-storage-config alone returns ValidationException. At least one of --node-recovery or --instance-groups must also be supplied. The approach is to read the current NodeRecovery value by running describe-cluster and pass it back unchanged. This has no side effect on the cluster configuration. InstanceMemoryAllocationPercentage accepts 20–100. Begin at 20 and increase as needed based on observed throughput and hit rate. Verify with [truncated for AI cost control]