翻訳待ち：Reduce ASR inference costs by 75% with NVIDIA MPS on Amazon EC2

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

This post is a collaboration between AWS, NVIDIA and Heidi. Reducing automatic speech recognition (ASR) inference costs on Amazon Elastic Compute Cloud (Amazon EC2) becomes critical when GPU utilization per request is low but latency requirements are strict. A single ASR inference request typically uses only 15–20 percent of a GPU’s compute capacity, yet the default time-slicing behavior in NVIDIA CUDA® forces sequential access, leaving 80 percent of the hardware idle. Heidi Health is an AI Care Partner that processes over 2.4 million clinical consultations per week across 190 countries. To sustain sub-second transcription latency at peak traffic, this inefficiency forces the company to run 16 GPU instances. In a previous post, you learned how to fine-tune a Nemotron speech model, NVIDIA Parakeet TDT 0.6B V2 for clinical speech recognition. In this post, we focus on what comes after fine-tuning: serving that model efficiently. We demonstrate how NVIDIA CUDA Multi-Process Service (MPS), combined with NVIDIA Triton Inference Server on Amazon EC2 GPU instances, reduces GPU infrastructure requirements by 75 percent (from 16 instances to 4). This setup maintains sub-second latency at 92.1 requests per second (RPS) per GPU. Solution overview This section covers the following: The GPU utilization challenge. The three available sharing mechanisms. Model-level optimizations with ONNX and TensorRT. Request scheduling with Triton. How these components integrate on Amazon EC2. The GPU utilization problem A single ASR inference request on the Parakeet TDT 0.6B V2 model uses roughly 15–20 percent of an NVIDIA L40S GPU’s 142 streaming multiprocessors (SMs). The remaining 80 percent sits idle during each forward pass. CUDA’s default time-slicing behavior compounds this waste by giving each process exclusive GPU access. Processes take turns, context switching adds overhead between them, and no concurrent execution occurs. The result: a single GPU handles only approximately 62 RPS at acceptable latency (mean Triton config ├── server.py # FastAPI gateway (OpenAI-compatible API) └── triton_model_repo/ └── parakeet_asr/ ├── config.pbtxt # Dynamic batching configuration └── 1/model.py # Python backend - direct forward pass Build and run the container Build the container image with your fine-tuned .nemo checkpoint as a build argument, then run with the MPS instance count you want. The Dockerfile bakes the checkpoint into the image and applies local attention optimization during the build step. # Build the all-in-one image docker build -f Dockerfile.single \ --build-arg LOCAL_NEMO_FILENAME=your_model.nemo \ -t parakeet-mps:latest . # Run with 4 MPS instances (25% SM each) on a single GPU docker run --gpus all --shm-size=2g \ -e MPS_INSTANCE_COUNT=4 \ -p 8002:8002 \ parakeet-mps:latest The container startup sequence is: (1) start the CUDA MPS daemon, (2) run auto_config.py to set the Triton instance count and SM percentage, (3) launch tritonserver. Refer to the repository for complete build and run instructions. Configure the deployment Environment variables (MPS_INSTANCE_COUNT, GATEWAY_WORKERS, TRITON_URL, CUDA_VISIBLE_DEVICES) control all runtime behavior. The same container image works across GPU types by changing MPS_INSTANCE_COUNT, which sets both the Triton instance group count and the SM percentage per instance. Refer to the repository README for the full configuration reference. Understand the key design decisions The implementation makes several important design choices that are critical for stable operation under CUDA MPS. We highlight the most important ones in this section. Direct forward pass. The Triton backend calls model.forward() directly instead of calling model.transcribe() through NeMo, removing approximately 50 ms of framework overhead per request. Combined with bfloat16 autocast and a dedicated CUDA stream per instance, a single instance processes 45-second audio in approximately 160 ms. Serialized model loading. Loading a 600M-parameter model from four processes simultaneously would exceed GPU memory. The backend serializes initialization using a file lock (fcntl.flock), with each instance loading, moving to GPU, freezing weights, and performing CUDA graph warmup before releasing the lock. CUDA graph warmup envelope. The TDT decoder uses CUDA graphs to eliminate kernel launch overhead. During initialization, the backend pre-warms all expected production shapes (5, 15, 30, 45, and 60 seconds at batch size 1, plus batch size 2 at 61 seconds). Shapes within this envelope replay cached graphs at approximately 165 ms. Shapes exceeding it fall back to eager execution (approximately 500 ms) for that call only. MPS-safe CUDA graph fallback. Under MPS, when NeMo’s decoder encounters a new tensor shape, it attempts to recapture the CUDA graph. During the 1.5–2.5 second recapture window, sibling MPS instances corrupt the capture, causing cudaErrorIllegalAddress and crashing the process. Wedge sentinel health monitoring. Under sustained load, CUDA errors in one MPS instance can leave it unrecoverable. When the backend detects a wedged instance (through a CUDA stream probe failure), it writes a sentinel file to tmpfs (/tmp/parakeet_wedged). Dynamic batching. Triton accumulates requests up to batch size 16 (preferred sizes 4, 8, 16) with a 50 ms max queue delay, balancing latency against throughput. Gateway-side audio decoding. The FastAPI gateway handles audio decoding (WAV, WebM/Opus, MP3, M4A, FLAC) using torchcodec, keeping the Triton input as raw float32 tensors so the gateway can run on CPU-only nodes. Streaming diarization The speaker diarization model (NVIDIA Streaming Sortformer 4-speaker v2) uses eight MPS instances at 12 percent SM each with sequence batching for per-recording state. Each recording gets a unique correlation ID for chunk routing, with sessions auto-expiring after 600 seconds. The model runs as a TensorRT + ONNX engine with warmup optimization at container start. API endpoints The gateway exposes an OpenAI Whisper-compatible API (POST /v1/audio/transcriptions), making it a drop-in replacement for existing integrations. Additional endpoints include /health (liveness + wedge sentinel check) and /metrics (Prometheus-format latency quantiles). Response formats include json, verbose_json, text, srt, and vtt. Refer to the repository for the full API reference. Results The benchmark sweeps concurrency from 1 to 100 on each configuration, averaging 5 rounds of measurements. Audio samples are representative clinical consultation segments. The SLA threshold is: mean latency 650 ms or p99 > 1,000 ms). The selected production path (Triton + MPS on g7e) achieves 92.1 RPS per GPU with 75% infrastructure savings Diarization results We benchmarked the diarization model before and after TensorRT engine warmup optimization: Metric Before Warmup After Warmup Improvement Mean 309.04 ms 238.73 ms -23% p50 348.21 ms 237.67 ms -32% p95 469.32 ms 355.82 ms -24% p99 499.45 ms 389.21 ms -22% The warmup optimization also reduced standard deviation from 12.62 ms to 7.13 ms (under 44 percent), indicating significantly more predictable inference latency. The model processes 60-second recordings in four chunks of 15 seconds each, all well within the overall pipeline budget. In operational terms, diarization processes a 60-second consultation in four chunks of 15 seconds at 238 ms mean latency per chunk (total under 1 second, real-time factor 0.016x). The eight diarization instances run on a separate MPS partition from transcription without contention. Clean up resources To avoid incurring ongoing charges after testing, clean up the resources you created while following this post: Stop and terminate the Amazon EC2 GPU instances (g6e.4xlarge or g7e.4xlarge). Delete attached Amazon EBS volumes (model checkpoints, TensorRT cache). Remove Docker images from Amazon ECR if pushed. Delete any Amazon CloudWatch log groups created during testing. Conclusion In this post, we showed how NVIDIA CUDA MPS on Amazon EC2 reduces ASR inference infrastructure by 75 percent (from 16 GPUs to 4) while maintaining sub-second latency SLAs (mean < 650 ms, p99 < 1,000 ms). On g7e.4xlarge, MPS achieves 92.1 RPS per GPU at 352 ms mean latency. The TensorRT + ONNX + MPS optimization pushes further to 111.6 RPS (88 percent reduction) for workloads where ONNX re-export on each fine-tuning cycle is acceptable. These optimizations are model-agnostic: the MPS architecture, direct forward-pass pattern, CUDA graph safety mechanism, and wedge sentinel apply to any encoder-decoder model served through Triton on NVIDIA GPUs. The same approach has been validated with NVIDIA Canary and OpenAI Whisper large-v3 checkpoints. The pattern extends to any workload where individual requests use a small fraction of available GPU compute. For production deployments, start with four MPS instances on g7e.4xlarge and monitor GPU SM utilization with nvidia-smi. If p99 latency has headroom, increase MPS_INSTANCE_COUNT incrementally. The TensorRT + ONNX + MPS path delivers an additional 21 percent throughput gain (111.6 compared to 92.1 RPS). The tradeoff is a longer deployment pipeline that requires weekly ONNX re-export. The accompanying GitHub repository contains the complete implementation: Dockerfiles, Triton model configurations, the FastAPI gateway, CUDA graph safety patch, health monitoring, and benchmark scripts, ready to deploy on any EC2 GPU instance. To get started, explore the following resources: Accompanying GitHub repository with complete implementation. Amazon EC2 G6e and G7e instances. NVIDIA Triton Inference Server documentation. Part 1: Fine-tuning NVIDIA NeMoTron Speech ASR on Amazon EC2 for domain adaptation. Acknowledgements The authors thank the following AWS and Heidi team members for their contributions to this post: Faisal Masood, Prem Oommen, Xuetong Wu, Taha Ansari, and Ocha Cakramurti. About the authors