SpecMD: A Comprehensive Study on Speculative Expert Prefetching
SpecMD is a standardized framework developed by Apple researchers to benchmark and evaluate expert caching policies for Mixture-of-Experts (MoE) models. The study reveals that MoE expert access patterns do not follow temporal locality, leading to the proposal of a new eviction policy called Least-Stale, which reduces collision misses by up to 85× compared to LRU and achieves 88% hit rates with 34.7% reduction in time-to-first-token on OLMoE.
SpecMD: A Comprehensive Study on Speculative Expert Prefetching - Apple Machine Learning Research
Machine Learning Research
Open MenuClose Menu
Overview
Research Highlights
Publications
Events
Work with us
research area Methods and Algorithms, research area Tools, Platforms, Frameworksconference ICML
content type paperpublished May 2026
SpecMD: A Comprehensive Study on Speculative Expert Prefetching
AuthorsDuc Hoang, Ajay Jaiswal, Mohammad Samragh Razlighi, Minsik Cho
View publication
Copy Bibtex
Mixture-of-Experts (MoE) models enable sparse expert activation, meaning that only a subset of the model’s parameters is used during each inference. However, to translate this sparsity into practical performance, an expert caching mechanism is required. Previous works have proposed hardware-centric caching policies, but how these various caching policies interact with each other and different hardware specification remains poorly understood. To address this gap, we develop SpecMD, a standardized framework for benchmarking ad-hoc cache policies on various hardware configurations. Using SpecMD, we perform an exhaustive benchmarking of several MoE caching strategies, reproducing and extending prior approaches in controlled settings with realistic constraints. Our experiments reveal that MoE expert access is not consistent with temporal locality assumptions (e.g LRU, LFU). Motivated by this observation, we propose Least-Stale, a novel eviction policy that exploits MoE’s predictable expert access patterns to reduce collision misses by up to 85× over LRU. With such gains, we achieve over 88% hit rates with up to 34.7% Time-to-first-token (TTFT) reduction on OLMoE at only 5% or 0.6GB of VRAM cache capacity.
Related readings and updates.
Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing
May 5, 2026research area Methods and Algorithms, research area Speech and Natural Language Processing
Serving transformer language models with high throughput requires caching Key-Values (KVs) to avoid redundant computation during autoregressive generation. The memory footprint of KV caching is significant and heavily impacts serving costs. This work proposes to lessen these memory requirements. While recent work has largely addressed KV cache reduction via compression and eviction along the temporal axis, we argue that the depth dimension offers…
Read more
MoEs Are Stronger than You Think: Hyper-Parallel Inference Scaling with RoE
January 12, 2026research area Data Science and Annotation, research area Speech and Natural Language Processing
The generation quality of large language models (LLMs) is often improved by utilizing inference-time sequence-level scaling methods (e.g., Chain-of-Thought). We introduce hyper-parallel scaling, a complementary framework that improves prediction quality at the token level. Hyper-parallel scaling computes and aggregates multiple output proposals for a single token from the model. We implement this concept in Mixture-of-Experts (MoE) models, which…
Read more
Discover opportunities in Machine Learning.
Our research in machine learning breaks new ground every day.
Work with us
Machine Learning Research
Publications
SpecMD: A Comprehensive Study on Speculative Expert Prefetching
Privacy Policy
Terms of Use
Legal
Copyright © 2026 Apple Inc. All rights reserved.