Apple Machine Learning Research

Research shows that LLM-as-a-judge panels suffer from correlated errors, drastically reducing their informational value. Tests with 9 frontier models from 7 families found only 2 effective independent votes, with accuracy 8-22 points lower than the ideal. The best single model matches or outperforms the full panel, and adding judges or using better aggregation helps little.

When annotators disagree on a label, the disagreement itself carries signal—and the number of annotators needed to capture it depends on the evaluation metric. We fine-tune NLI models on label distributions subsampled from ChaosNLI and find that entropy correlation requires 20-50 annotators to converge, while KL divergence saturates at 10. Soft labels capture item-specific signal that label smoothing cannot replicate.

Apple announces its third generation of Foundation Models, a family of five models built with Google, including on-device and server-based models, with a focus on privacy and new architectures like sparsely activated models and instruction-following pruning. The models power new Siri and intelligent tools, and show significant quality improvements in evaluations.

Apple is showcasing new research at the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2026 in Denver, June 3-7. The company is sponsoring the conference and presenting work on video generation, multimodal understanding, image compression, and more.

Streaming vision-language models (VLMs) continuously generate responses from an instruction and an online frame stream, crucial for real-time visual assistants. Existing benchmarks focus on offline evaluation, missing metrics like proactiveness and consistency. VSAS-Bench introduces a new framework with over 18,000 dense annotations, synchronous/asynchronous protocols, and reveals that conventional VLMs can be adapted without extra training to outperform dedicated streaming models.

Modern large language models (LLMs) extend context lengths to millions of tokens, enabling coherent, personalized responses grounded in long conversational history. However, the Key-Value (KV) cache grows linearly with the extended dialogue history, causing the model’s memory footprint to quickly exceed device limits. While recent KV cache compression methods attempt to reduce memory usage, most apply cache eviction after processing the entire context, incurring unbounded peak memory usage. Additionally, query-dependent eviction narrows the cache semantics to a single query, leading to failure cases in multi-turn conversations. In this paper, we introduce EpiCache, a training-free KV cache management framework for long conversational question answering (LongConvQA) under fixed memory budgets. EpiCache bounds cache growth through block-wise prefill and preserves topic-relevant context via episodic KV compression, which clusters conversation history into coherent episodes and performs episode-specific KV cache eviction. Across three LongConvQA benchmarks (LongMemEval, Realtalk, and LoCoMo), EpiCache improves accuracy by up to 30%, achieves near full-cache accuracy under 4-6x compression, and reduces latency and peak memory by up to 2.4x and 3.7x, respectively.

Apple researchers propose BalCapRL, a balanced reinforcement learning framework that jointly optimizes correctness, coverage, and linguistic quality for MLLM image captioning. By introducing GDPO-style reward-decoupled normalization and length-conditional reward masking, BalCapRL achieves significant gains: +13.6 DCScore, +9.0 CaptionQA, and +29.0 CapArena across LLaVA-1.5 and Qwen2.5-VL models.

Current critic-less RLHF methods aggregate multi-objective rewards via an arithmetic mean, leaving them vulnerable to constraint neglect: high-magnitude success in one objective can numerically offset critical failures in others (e.g., safety or formatting), masking low-performing “bottleneck” rewards vital for reliable multi-objective alignment. We propose Reward-Variance Policy Optimization (RVPO), a risk-sensitive framework that penalizes inter-reward variance during advantage aggregation, shifting the objective from “maximize sum” to “maximize consistency.” We show via Taylor expansion that a LogSumExp (SoftMin) operator effectively acts as a smooth variance penalty. We evaluate RVPO on rubric-based medical and scientific reasoning with up to 17 concurrent LLM-judged reward signals (Qwen2.5-3B/7B/14B) and on tool-calling with rule-based constraints (Qwen2.5-1.5B/3B). By preventing the model from neglecting difficult constraints to exploit easier objectives, RVPO improves overall scores on HealthBench (0.261 vs. 0.215 for GDPO at 14B, p < 0.001) and maintains competitive accuracy on GPQA-Diamond without the late-stage degradation observed in other multi-reward methods, demonstrating that variance regularization mitigates constraint neglect across model scales without sacrificing general capabilities.

Velox is a framework for learning latent representations of 4D objects from unstructured dynamic point clouds. It compresses spatiotemporal color point clouds into dynamic shape tokens, supervised by a 4D surface decoder for geometry and a Gaussian decoder for appearance. Strong performance is demonstrated on video-to-4D generation, 3D tracking, and cloth simulation.

Apple hosted a two-day workshop in early 2026 focusing on privacy-preserving machine learning and AI, bringing together researchers to discuss advances in federated learning, foundation model privacy, attacks and security, and more.

We propose HeadsUp, a scalable feed-forward method for reconstructing high-quality 3D Gaussian heads from large-scale multi-camera setups. Our method employs an efficient encoder-decoder architecture that compresses input views into a compact latent representation, which is then decoded into UV-parameterized 3D Gaussians anchored to a neutral head template. This UV representation decouples the number of 3D Gaussians from the number and resolution of input images, enabling training with many high-resolution views. Trained on an internal dataset of over 10,000 subjects—an order of magnitude larger than existing datasets—HeadsUp achieves state-of-the-art reconstruction quality and generalizes to novel identities without test-time optimization. We analyze scaling behavior and demonstrate downstream applications for novel identity generation and expression-based animation.

Apple ML research proposes Text-Conditional JEPA (TC-JEPA), which uses image captions to reduce prediction uncertainty in masked feature prediction, leading to more semantically meaningful visual representations. It outperforms contrastive methods on diverse tasks, especially those requiring fine-grained understanding and reasoning.

Apple ML Research presents a paper at CVPR 2026 systematically studying key modeling choices for practical learned image codecs jointly optimized for perceptual quality and runtime. Using performance-aware neural architecture search, they build a new codec that achieves 2.3–3x bitrate savings over traditional codecs like AV1 and VVC, and 20–40% over learned alternatives, with encoding/decoding speeds of 230ms/150ms on iPhone 17 Pro Max for 12MP images.

SpecMD is a standardized framework developed by Apple researchers to benchmark and evaluate expert caching policies for Mixture-of-Experts (MoE) models. The study reveals that MoE expert access patterns do not follow temporal locality, leading to the proposal of a new eviction policy called Least-Stale, which reduces collision misses by up to 85× compared to LRU and achieves 88% hit rates with 34.7% reduction in time-to-first-token on OLMoE.

Apple ML Research introduces iTARFlow, an iterative denoising approach that enhances Normalizing Flows for image generation, achieving competitive results on ImageNet at multiple resolutions.

Apple ML Research introduces SFI-Bench, a video-based benchmark with over 1,700 questions to evaluate multimodal LLMs on spatial and functional reasoning. Experiments show current models struggle to integrate spatial memory with functional knowledge, highlighting a critical bottleneck.

Apple ML Research proposes Stochastic KV Routing, a training technique where layers randomly attend to their own or a preceding layer's KV states, enabling adaptive depth-wise cache sharing. This reduces KV cache memory footprint without throughput loss and can even improve performance in data-constrained settings.

Apple and Purdue University propose PORTool, a policy optimization algorithm that uses a rewarded rollout tree and step-level importance estimates to address credit assignment in multi-tool reasoning, improving accuracy while reducing tool calls.

Apple researchers propose an inference-time evaluation method that introduces a reviewer agent to assess provisional tool calls before execution, enabling real-time error correction. Evaluated on BFCL and τ2-Bench, the method achieves +5.5% and +7.1% improvements, and introduces Helpfulness-Harmfulness metrics to quantify the tradeoff of corrections.

Apple is presenting new research at the annual ICASSP, taking place in Barcelona, Spain, from May 4 to 8, 2026. The company is sponsoring the conference and will showcase three papers covering audio-visual speech models, spatial audio generation, and speculative decoding.

Researchers at Apple and Gallaudet University developed a pseudo-annotation pipeline to address the scarcity of high-quality annotated sign language data. Their approach uses a fingerspelling recognizer, isolated sign recognizer, and a K-Shot LLM to generate likely annotations from signed video and English input. They achieved state-of-the-art results on FSBoard (6.7% CER) and ASL Citizen (74% top-1 accuracy) and are releasing nearly 500 human-annotated videos and over 300 hours of pseudo-annotations.

Apple ML Research presents STARFlow-V, a normalizing flow-based video generator offering end-to-end learning, robust causal prediction, and native likelihood estimation. It uses a global-local architecture in spatiotemporal latent space, flow-score matching, and video-aware Jacobi iteration to achieve strong visual fidelity and temporal consistency, demonstrating the first evidence that normalizing flows can produce high-quality autoregressive video generation.

Apple ML Research introduces DSO (Direct Steering Optimization), a reinforcement learning-based method that learns linear transformations to steer model activations, effectively mitigating bias in VLMs and LLMs. It achieves state-of-the-art fairness-capabilities trade-off with inference-time control.

Apple Machine Learning Research introduces Sonata, a lightweight adapter that uses self-consistency prediction to dynamically allocate thinking budgets during inference, reducing thinking tokens by 20-80% while maintaining accuracy, or improving accuracy by up to 5% with the same token cost.

LaDiR combines VAE and latent diffusion to enable iterative refinement of reasoning trajectories, improving accuracy, diversity, and interpretability over autoregressive methods.

Apple researchers present StereoFoley, a framework that generates semantically aligned, temporally synchronized, and spatially accurate stereo sound at 48 kHz from video. Using a synthetic data pipeline and fine-tuning, it achieves object-aware stereo imaging.

The paper investigates how conditional diffusion models achieve compositional generalization, specifically length generalization—generating images with more objects than seen during training. Experiments on CLEVR show success in some cases but not all, pointing to the importance of local conditional scores. The authors prove an equivalence between compositional structure and local conditional scores, and demonstrate that enforcing locality enables generalization in failing models. Analysis of SDXL reveals spatial locality but absent conditional locality in pixel space, yet evidence of local conditional scores in feature space.