AVI-HT: Adaptive Vision-IMU Fusion for 3D Hand Tracking
AVI-HT is an adaptive visual-IMU fusion approach for 3D hand pose tracking that jointly models egocentric images with on-glove 6-DoF IMU signals. It achieves significantly improved accuracy and availability, especially in hand-object interaction scenarios with heavy visual occlusion. Key ingredients include synchronized multi-modal training data and a cross-sensor deep attention mechanism. Experiments on the DexGloveHOI dataset show that AVI-HT reduces mean keypoint error by 16.1% and its wrist-aligned variant by 24.2% over baselines.
Article intelligence
Key points
- AVI-HT fuses vision and IMU signals for accurate 3D hand tracking
- Cross-sensor attention adaptively modulates trust in vision and IMU sensors
- Reduces mean keypoint error by 16.1% on a large-scale dataset
- Ablation studies reveal per-finger IMU contributions across activities
Why it matters
This matters because AVI-HT fuses vision and IMU signals for accurate 3D hand tracking.
Technical impact
May affect model selection, inference cost, product capability, and evaluation benchmarks.
[2605.21714] AVI-HT: Adaptive Vision-IMU Fusion for 3D Hand Tracking
[Submitted on 20 May 2026]
Title:AVI-HT: Adaptive Vision-IMU Fusion for 3D Hand Tracking
View a PDF of the paper titled AVI-HT: Adaptive Vision-IMU Fusion for 3D Hand Tracking, by Ziyi Kou and 6 other authors
View PDF HTML (experimental)
Abstract:We present AVI-HT, an adaptive visual-IMU fusion approach for tracking 3D hand poses by jointly modeling the egocentric image with on-glove 6-DoF IMU signals. AVI-HT achieves significantly improved accuracy and availability, particularly in hand-object interaction (HOI) scenarios involving heavy visual occlusion. Two complementary ingredients underpin its success: (1) synchronized multi-modal training data pairing on-body vision-IMU sensor streams with ground-truth 3D hand poses from a motion-capture system, and (2) a cross-sensor deep attention mechanism that adaptively modulates the trust assigned to the vision and individual IMU sensors. To evaluate AVI-HT in real-world settings, we conduct extensive experiments on our DexGloveHOI dataset that consists of 100K+ pairwise vision-IMU samples with synchronized 3D annotated poses, in which users manipulate a variety of objects during daily tasks. We compare against multiple single- and multi-modal tracking approaches under two hand models (UmeTrack, MANO). The results show that AVI-HT reduces mean keypoint error by 16.1% and its wrist-aligned variant by 24.2% over the baselines. Ablation studies further reveal the per-finger contribution of IMU sensors across activity types, and the model's sensitivity to IMU noise and temporal misalignment in vision-IMU fusion.
Subjects:
Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)
Cite as: arXiv:2605.21714 [cs.CV]
(or arXiv:2605.21714v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2605.21714
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Ziyi Kou [view email] [v1] Wed, 20 May 2026 20:19:24 UTC (5,653 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled AVI-HT: Adaptive Vision-IMU Fusion for 3D Hand Tracking, by Ziyi Kou and 6 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CV
new | recent | 2026-05
Change to browse by:
cs cs.RO
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)