Modal Blog

Maximizing perf on AI-SQL queries with the KV-optimal left-deep join

Learn how we optimized performance and efficiency serving the workload that is changing software engineering forever — and you can too.

Recent product updates from Modal and news from around the community.

Botika builds agentic e-commerce tools for fashion brands and runs its entire AI stack on Modal: a 100-terabyte data pipeline, custom foundation-model research, and about 15 production inference models. CEO Eran Dagan says Modal removed the cloud-engineering overhead that once slowed them down and lets the team scale traffic dramatically without operational burden.

Modal is opening a London office as part of a larger European investment. The company already has a Stockholm engineering team of about 20 people and now expands to London to build GTM teams, with plans to grow engineering and other functions later. European AI companies like Black Forest Labs and Legora, as well as global platforms like DoorDash, are using Modal.

Qwen3.8-2.4T-A95B by Alibaba, with a 1M token context window, is now available via Modal Auto Endpoints.

Modal’s Function Call data path is now >50ms faster. Our new routing layer is geographically distributed, so you can further reduce your network overhead.

Hugging Face published a technical timeline of a recent agent intrusion, naming Modal as the third-party infrastructure used. Modal confirmed its platform and isolation were not compromised.

Moonshot has released Kimi K3, a 2.8 trillion parameter multimodal model, now available on Modal at 460 tokens per second. The model features a mixture-of-experts architecture, 1M token context window, native vision, and is optimized with a custom DFlash speculator for faster inference.

Cognition's Devin, an AI software engineer, can now run in Modal sandboxes via Devin Outposts, enabling customized environments with GPU support and fast cold-starts.

Thinking Machines has released Inkling, a general-purpose multimodal model accepting text, image, and audio inputs, now available on Modal as a Managed Endpoint with token-based pricing. The post details its architecture, including local attention and DFlash speculation for fast inference.

To compare rates for serverless and reserved GPUs, look at your application's peak-to-average ratio.

Multi-token Residual Prediction (MRP) is a lightweight module for diffusion language models that predicts the residual between adjacent denoising steps rather than the full distribution. This enables both lossless speedup in static regimes (up to 1.56×) and quality recovery in dynamic regimes (up to +16 accuracy points) without trade-offs.

Anthropic launches Claude Science, an AI workbench for life sciences researchers, integrated with Modal to provide elastic compute infrastructure for running data processing, structure prediction, and molecule design directly from a conversation.

A deep dive inside Modal's new ultra-low-latency serverless server product, explaining the architecture decisions behind building a custom proxy (fprs) using Pingora, Envoy at the edge, and Spanner for configuration, all optimized for LLM inference workloads.

Modal and Decagon collaborated to cut inference latency by 100ms using speculative decoding, outperforming proprietary providers. The article details the low-latency playbook including optimization of communication, host overhead, prefill, and decode latencies, with a focus on custom speculative decoding models (DFlash) for big wins.

Modal launches Auto Endpoints, a self-serve on-ramp to production-grade LLM inference, allowing users to deploy frontier open models with a single command and gain full visibility and control over inference code, metrics, and infrastructure. Built on Modal's AI infrastructure platform, it features high-performance autoscaling, custom container runtime, global GPU availability, and Modal Servers for ultra-low-latency routing (5ms overhead). Pre-tuned recipes from top-tier team experience and DFlash speculative decoding are included. Future roadmap includes full automation of inference engineering.

Modal is all-in on speculative decoding, arguing it's the single most important inference optimization, delivering 2-3x speedups. They released state-of-the-art DFlash speculators for Qwen models, achieving 5-20% extra speedups, and explain the theory, simulation, and math behind the acceleration.

This article explores the practical application of reinforcement learning in post-training large language models, highlighting that the current bottleneck is infrastructure rather than algorithms. Modal shares its experience running RL post-training at scale and introduces its open-source library to help teams address key challenges like multi-node training, environment management, and GPU utilization.

Modal introduces Role-Based Access Control for all Team and Enterprise users, built on Environments to provide granular permissions for humans and AI agents.

Modal raised $355M at a $4.65B valuation, led by General Catalyst and Redpoint. The company has grown fivefold since September, exceeding $300M in annualized revenue. Modal provides a cloud platform for AI, focusing on elastic inference, agent runtimes, and sandboxes. The funding will support expansion in low-latency inference, reinforcement learning, and agent compute.

Applied Compute trains custom AI agents for enterprises using Reinforcement Learning, focusing on post-training to differentiate from commoditized frontier models. Their 'Specific Intelligence' approach leverages Modal's infrastructure for fast, flexible, and reliable RL training loops serving clients like DoorDash, Cognition, and Mercor.

Anthropic and Modal announce the integration of Claude Managed Agents with Modal Sandboxes, allowing developers to run tool calls in self-hosted, customizable sandboxes with fast startup, cost efficiency, and scalability. The collaboration enables secure, isolated execution environments for AI agents, with early adopters like Mason AI, DoorDash, and Blend sharing positive experiences.

Modal's deep engineering reduces GPU inference server boot times from kiloseconds to tens of seconds, enabling truly serverless computing for variable inference workloads.

By profiling SGLang's scheduler, Modal engineers discovered a bottleneck from repeated CUDA IPC pool handle opens. Replacing them with a simple Python dictionary cache improved throughput by 16.2% and reduced latency by over 10% on Qwen2.5-VL-3B. The optimization is merged in SGLang v0.5.10.

AE Studio used Modal to compare Evolution Strategies and GRPO for training LLMs on math theorem proving. By leveraging Modal's parallel GPUs, sandboxed verification, and volume storage, they reduced setup time by 60% and cost by up to 75%. Early results show ES matching or outperforming GRPO in several scenarios, especially with limited data.

Modal becomes an official sandbox provider for the OpenAI Agents SDK. This article demonstrates how to build a custom coding agent harness from scratch, integrating Modal sandboxes for secure, parallel, and scalable automation, using the Parameter Golf challenge as an example.

Modal integrates with Autoresearch to provide elastic GPU scaling, allowing AI agents to dynamically provision compute resources. In a Parameter Golf challenge, an agent ran 113 experiments across 238 GPU-hours, achieving 5x speedup over a single workstation while using a fraction of a dedicated cluster's resources.

Modal announces that Butter, an AI sandbox technology company, is joining Modal. Founder Erik Dunteman and researcher Raymond Tana will join the Modal Sandbox team. Butter's expertise includes agent harness engineering and the development of bVisor, a lightweight ephemeral sandbox built with Zig.

Physical Intelligence uses Modal to achieve low-latency remote real-time inference for robots, with a specialized QUIC-based transport adding only 10-15 ms of network overhead, enabling experimentation with larger models.