Modal Blog

A deep dive inside Modal's new ultra-low-latency serverless server product, explaining the architecture decisions behind building a custom proxy (fprs) using Pingora, Envoy at the edge, and Spanner for configuration, all optimized for LLM inference workloads.

Modal and Decagon collaborated to cut inference latency by 100ms using speculative decoding, outperforming proprietary providers. The article details the low-latency playbook including optimization of communication, host overhead, prefill, and decode latencies, with a focus on custom speculative decoding models (DFlash) for big wins.

Modal launches Auto Endpoints, a self-serve on-ramp to production-grade LLM inference, allowing users to deploy frontier open models with a single command and gain full visibility and control over inference code, metrics, and infrastructure. Built on Modal's AI infrastructure platform, it features high-performance autoscaling, custom container runtime, global GPU availability, and Modal Servers for ultra-low-latency routing (5ms overhead). Pre-tuned recipes from top-tier team experience and DFlash speculative decoding are included. Future roadmap includes full automation of inference engineering.

Modal is all-in on speculative decoding, arguing it's the single most important inference optimization, delivering 2-3x speedups. They released state-of-the-art DFlash speculators for Qwen models, achieving 5-20% extra speedups, and explain the theory, simulation, and math behind the acceleration.

This article explores the practical application of reinforcement learning in post-training large language models, highlighting that the current bottleneck is infrastructure rather than algorithms. Modal shares its experience running RL post-training at scale and introduces its open-source library to help teams address key challenges like multi-node training, environment management, and GPU utilization.

Modal introduces Role-Based Access Control for all Team and Enterprise users, built on Environments to provide granular permissions for humans and AI agents.

Modal raised $355M at a $4.65B valuation, led by General Catalyst and Redpoint. The company has grown fivefold since September, exceeding $300M in annualized revenue. Modal provides a cloud platform for AI, focusing on elastic inference, agent runtimes, and sandboxes. The funding will support expansion in low-latency inference, reinforcement learning, and agent compute.

Applied Compute trains custom AI agents for enterprises using Reinforcement Learning, focusing on post-training to differentiate from commoditized frontier models. Their 'Specific Intelligence' approach leverages Modal's infrastructure for fast, flexible, and reliable RL training loops serving clients like DoorDash, Cognition, and Mercor.

Anthropic and Modal announce the integration of Claude Managed Agents with Modal Sandboxes, allowing developers to run tool calls in self-hosted, customizable sandboxes with fast startup, cost efficiency, and scalability. The collaboration enables secure, isolated execution environments for AI agents, with early adopters like Mason AI, DoorDash, and Blend sharing positive experiences.

Modal's deep engineering reduces GPU inference server boot times from kiloseconds to tens of seconds, enabling truly serverless computing for variable inference workloads.

By profiling SGLang's scheduler, Modal engineers discovered a bottleneck from repeated CUDA IPC pool handle opens. Replacing them with a simple Python dictionary cache improved throughput by 16.2% and reduced latency by over 10% on Qwen2.5-VL-3B. The optimization is merged in SGLang v0.5.10.

AE Studio used Modal to compare Evolution Strategies and GRPO for training LLMs on math theorem proving. By leveraging Modal's parallel GPUs, sandboxed verification, and volume storage, they reduced setup time by 60% and cost by up to 75%. Early results show ES matching or outperforming GRPO in several scenarios, especially with limited data.

Modal becomes an official sandbox provider for the OpenAI Agents SDK. This article demonstrates how to build a custom coding agent harness from scratch, integrating Modal sandboxes for secure, parallel, and scalable automation, using the Parameter Golf challenge as an example.

Modal integrates with Autoresearch to provide elastic GPU scaling, allowing AI agents to dynamically provision compute resources. In a Parameter Golf challenge, an agent ran 113 experiments across 238 GPU-hours, achieving 5x speedup over a single workstation while using a fraction of a dedicated cluster's resources.

Modal announces that Butter, an AI sandbox technology company, is joining Modal. Founder Erik Dunteman and researcher Raymond Tana will join the Modal Sandbox team. Butter's expertise includes agent harness engineering and the development of bVisor, a lightweight ephemeral sandbox built with Zig.

Physical Intelligence uses Modal to achieve low-latency remote real-time inference for robots, with a specialized QUIC-based transport adding only 10-15 ms of network overhead, enabling experimentation with larger models.

Modal announces product updates including NVIDIA RTX Pro 6000 Blackwell GPU support, Command K palette in dashboard, Sandbox Filesystem API Beta, SDK improvements, and customer stories.

Runway partners with Modal to power real-time inference for Runway Characters, a real-time video agent API that generates expressive digital personas from a single image. Modal's serverless platform enables low-latency, multi-GPU inference globally, allowing Runway to move from proof of concept to production in under 30 days.

Doppel, an AI-native cybersecurity platform, migrated its ML workflows to Modal to accelerate experimentation and simplify inference deployment. Training experiments became parallel, reducing feedback loops. Inference builds dropped from 30 minutes to under a minute, with automatic scaling for traffic spikes.

A roundup of everything we shipped in February: Directory Snapshots for Sandboxes, a free GLM-5 endpoint, new billing API, and more.