StereoFoley: Object-Aware Stereo Audio Generation from Video - Apple Machine Learning Research
Machine Learning Research
Open MenuClose Menu
Overview
Research Highlights
Publications
Events
Work with us
research area Tools, Platforms, Frameworksconference ICASSP
content type paperpublished April 2026
StereoFoley: Object-Aware Stereo Audio Generation from Video
AuthorsTornike Karchkhadze†**, Kuan-Lin Chen, Mojtaba Heydari, Robert Henzel, Alessandro Toso, Mehrez Souden, Joshua Atkins
View publication
Copy Bibtex
We present StereoFoley, a video-to-audio generation framework that produces semantically aligned, temporally synchronized, and spatially accurate stereo sound at 48 kHz. While recent generative video-to-audio models achieve strong semantic and temporal fidelity, they largely remain limited to mono or fail to deliver object-aware stereo imaging, constrained by the lack of professionally mixed, spatially accurate video-to-audio datasets. First, we develop a base model that generates stereo audio from video, achieving performance on par with state-of-the-art V2A models in both semantic accuracy and synchronization. Next, to overcome dataset limitations, we introduce a synthetic data generation pipeline that combines video analysis, object tracking, and audio synthesis with dynamic panning and distance-based loudness controls, enabling spatially accurate object-aware sound. Finally, we fine-tune the base model on this synthetic dataset, yielding clear object-audio correspondence. Since no established metrics exist, we introduce a stereo object-awareness metric and report it alongside a human listening study; the two evaluations exhibit consistent trends. This work establishes the first end-to-end framework for stereo object-aware video-to-audio generation, addressing a critical gap in the field.
† UC San Diego
** Work done while at Apple
Related readings and updates.
ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Model
February 12, 2025research area Human-Computer Interaction, research area Speech and Natural Language Processingconference ICASSP
We introduce ImmerseDiffusion, an end-to-end generative audio model that produces 3D immersive soundscapes conditioned on the spatial, temporal, and environmental conditions of sound objects. ImmerseDiffusion is trained to generate first-order ambisonics (FOA) audio, which is a conventional spatial audio format comprising four channels that can be rendered to multichannel spatial output. The proposed generative system is composed of a spatial…
Read more
Learning Spatially-Aware Language and Audio Embeddings
December 9, 2024research area Methods and Algorithms, research area Speech and Natural Language Processingconference NeurIPS
Humans can picture a sound scene given an imprecise natural language description. For example, it is easy to imagine an acoustic environment given a phrase like “the lion roar came from right behind me!”. For a machine to have the same degree of comprehension, the machine must know what a lion is (semantic attribute), what the concept of “behind” is (spatial attribute) and how these pieces of linguistic information align with the semantic and…
Read more
Discover opportunities in Machine Learning.
Our research in machine learning breaks new ground every day.
Work with us
Machine Learning Research
Publications
StereoFoley: Object-Aware Stereo Audio Generation from Video
Privacy Policy
Terms of Use
Legal
Copyright © 2026 Apple Inc. All rights reserved.