Systematic Optimization of Real-Time Diffusion Model Inference on Apple M3 Ultra
This study conducts comprehensive 10-phase optimization experiments on Apple M3 Ultra (60-core GPU, 512 GB unified memory) to achieve real-time camera img2img transformation. By combining CoreML conversion of the distillation-specialized model SDXS-512 with a 3-thread camera pipeline, it reaches 22.7 FPS at 512x512 resolution. The work demonstrates that CUDA optimization insights do not transfer to Apple Silicon's unified memory architecture, with quantization showing no speedup, parallel inference being ineffective, and the Neural Engine unsuitable for large models, providing practical guidelines for Apple Silicon diffusion model inference.
Article intelligence
Key points
- 10-phase systematic optimization on Apple M3 Ultra using techniques like CoreML, quantization, Token Merging, and Neural Engine.
- Achieved 22.7 FPS real-time img2img at 512x512 with CoreML-converted SDXS-512 and 3-thread pipeline.
- Found CUDA insights ineffective on Apple Silicon: no speedup from quantization, parallel inference ineffective, Neural Engine unsuitable for large models.
- Provides practical guidelines for diffusion model inference on Apple Silicon.
Why it matters
This matters because 10-phase systematic optimization on Apple M3 Ultra using techniques like CoreML, quantization, Token Merging, and Neural Engine.
Technical impact
May affect model selection, inference cost, product capability, and evaluation benchmarks.
[2605.16259] Systematic Optimization of Real-Time Diffusion Model Inference on Apple M3 Ultra
[Submitted on 10 Feb 2026]
Title:Systematic Optimization of Real-Time Diffusion Model Inference on Apple M3 Ultra
View a PDF of the paper titled Systematic Optimization of Real-Time Diffusion Model Inference on Apple M3 Ultra, by Yoichi Ochiai
View PDF HTML (experimental)
Abstract:While real-time image generation using diffusion models has advanced rapidly on NVIDIA GPUs, systematic optimization research on non-CUDA platforms such as Apple Silicon remains extremely limited. In this study, we conducted comprehensive optimization experiments across 10 phases targeting the Apple M3 Ultra (60-core GPU, 512 GB unified memory) with the goal of achieving real-time camera img2img transformation. We explored a wide range of techniques including CoreML conversion, quantization, Token Merging, Neural Engine utilization, compact model exploration, frame interpolation, kNN search-based synthesis, pix2pix-turbo, optical flow frame skipping, and knowledge distillation, quantitatively evaluating the effectiveness of each approach. Ultimately, by combining CoreML conversion of the distillation-specialized model SDXS-512 with a 3-thread camera pipeline, we achieved real-time camera img2img transformation at 22.7 FPS at 512x512 resolution. The primary contribution of this work is the systematic demonstration that optimization insights established for CUDA are not necessarily effective on Apple Silicon's unified memory architecture. We reveal an optimization landscape fundamentally different from that of NVIDIA GPUs -- including the absence of speedup from quantization, the ineffectiveness of parallel inference, and the unsuitability of the Neural Engine for large-scale models -- and provide practical guidelines for diffusion model inference on Apple Silicon.
Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)
Cite as: arXiv:2605.16259 [cs.LG]
(or arXiv:2605.16259v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.16259
arXiv-issued DOI via DataCite
Submission history
From: Yoichi Ochiai Prof. [view email] [v1] Tue, 10 Feb 2026 20:40:45 UTC (17 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Systematic Optimization of Real-Time Diffusion Model Inference on Apple M3 Ultra, by Yoichi Ochiai
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.LG
new | recent | 2026-05
Change to browse by:
cs cs.AI cs.DC
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
IArxiv recommender toggle
IArxiv Recommender (What is IArxiv?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)