翻訳待ち：Bringing serverless functions closer to the speed of wire

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

All posts Back Engineering August 4, 2026 •6 minute read Bringing serverless functions closer to the speed of wire Daniel Shaar Member of Technical Staff @dshaar_ Function Call network hops through our legacy us-east I/O servers (left) vs. our new regional I/O servers (right). On Modal, you’ve always had control over which region your containers run in. However, every Function’s inputs and outputs would always get routed through our servers (a.k.a. the I/O plane) in us-east—potentially adding up to a few 100ms of network latency to each Function Call depending on the caller and container regions. Say you’re running a tiny embedding model for your RAG app that has str: return f"Hello, {name}" This week, we’ve completed the migration of all Functions running on Modal to use the new I/O plane in us-east. End-to-end Function Call latencies have dropped by roughly 80ms at p50. Enjoy the speed up! Life of a Function Call When you call .remote() on a Modal Function, your client serializes the input payload and sends it to our I/O plane. The client then immediately polls the I/O plane, waiting some amount of time each poll for the serialized output to become available. Once fetched, the client deserializes the output payload. Meanwhile, the I/O plane stores and queues the input in a pending state in Redis. Containers with available capacity (as defined by their input concurrency) constantly fetch inputs for processing from the queue, marking them as running. Once on the container, the input is deserialized and processed by the Function code. When the processing is done, the output is serialized and written to Redis, and the input is removed from the queue. Since the client is constantly communicating with the I/O plane by polling for the output, if something goes wrong with processing (i.e. server failure, container failure, or retriable user code failure), the client will find out quickly and automatically retry the input. Off the hot path of input processing: The I/O plane sends autoscaling information (e.g. running inputs, input backlog size, average execution time, etc.) about each Function to Modal’s control plane. This feeds into our scheduler, which uses the data to determine how many containers should be brought up or spun down. The I/O plane reports input processing events to power the dashboard and o11y pipelines. These events include when an input was received by the I/O plane, assigned to a container, began executing, etc. Containers heartbeat against the control plane to ensure healthy operation and against the I/O plane to keep track of inputs. Optimizing Function performance The I/O system has undergone a complete redesign informed by our experience operating Modal Functions at significant scale—each Function can support thousands of containers processing hundreds of inputs every second. Small wins add up From a systems design perspective, there were two major shifts in our code: All non-critical work (updating autoscaling stats, publishing input events to power the dashboard, etc.) now happens asynchronously in the background. Interactions with shared storage on the hot path of a Function Call are minimized (as our CEO Erik always says - Every Millisecond Counts). While this sounds simple in theory, doing this requires everything from client-side metadata caching to a complete redesign of our auth model—the client now receives and refreshes JWTs for faster auth. E2E request latency within us-east On the technology side, the most impactful change we made to our server layer was rewriting it fully in Go. When it comes to building highly concurrent gRPC servers, Go’s lightweight concurrency model makes it a natural choice over Python. Async Python is cooperatively single-threaded, which makes performance susceptible to event loop lag—a common pain point when handling CPU-intensive operations (e.g. writing large payloads over network). Another technology choice we revisited was the engine backing our Elasticache deployments—namely Redis OSS vs. Valkey. Valkey is forked off of Redis 7.2 and uses I/O multithreading to get more utilization out of the single execution thread by moving the I/O work to separate threads. Engine CPU usage for a representative intensive workload. Our input queues are built on top of Redis streams. During our input queue load testing, we observed that engine CPU spikes aggressively on Redis 7.2 onwards. As a result, we’ve stuck with Redis 7.1—for the curious, here’s the culprit change. Best user practices Now that we’ve talked a bit about how we’ve gotten the most for Modal, here’s how you can get the most out of Modal. Once you’ve made your execution times faster, the next place to look is network overhead. If your clients are located in a particular region, select the routing_region closest to them. You can optionally restrict your container region to be near the routing_region as well; however, there is a tradeoff between geographic restriction and available capacity to be mindful of (as well as cost implications). For inputs larger than 2MiB, the client will first upload the payload to our blob storage. Then, the client will send over the blob ID instead of the payload. Performance sensitive workloads should aim to keep the input and output payload sizes under 2MiB to avoid these extra network trips. One final trick to consider is whether inputs can be batched together in the client. If you have many small and fast inputs, you can reduce the number of network hops needed for them to land on a container by batching them into one larger input: @app.function() @modal.concurrent(max_inputs=100) def square(x: int) -> int: return x 2 @app.function() def batched_square(xs: list[int]) -> list[int]: return [x 2 for x in xs] @app.local_entrypoint() async def main(): slow_squares = await asyncio.gather( *[square.remote.aio(i) for i in range(100)] ) fast_squares = await batched_square.remote.aio(list(range(100))) Fin Modal Functions are the foundational building blocks for autoscaling serverless compute. With our new I/O plane, we’ve questioned every aspect of their internal operation to achieve significant reliability and performance wins. It took a huge effort from our Core Services I/O team to build a migration that could seamlessly migrate our users to the new system. We set out to deliver a big win without causing user toil, and we had to bridge many incompatibilities between the two architectures to do so. Special shout out to Egor Gagushin, Sam Ross, Parthiv Apsani, and Deven Navani for the marathon grind. Parthiv (left) and Egor (right) holding up our team motto the day we completed the migration. If you’re interested in building SoTA infra and making it super performant, we’re hiring!