Modal and Decagon collaborated to cut inference latency by 100ms using speculative decoding, outperforming proprietary providers. The article details the low-latency playbook including optimization of communication, host overhead, prefill, and decode latencies, with a focus on custom speculative decoding models (DFlash) for big wins.

Most retrieval systems use a single pipeline for all queries, which either under-serves hard queries or wastes compute on easy ones. This article presents cheap signals—like score spread and retriever agreement—to detect weak retrieval without an LLM, enabling selective escalation only when needed.