翻訳待ち：AgentHands: Generating interactive hand gestures for spatially grounded agent conversations in XR

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

AgentHands: Generating interactive hand gestures for spatially grounded agent conversations in XR August 25, 2026 Xun Qian, Research Scientist, and Ruofei Du, Interactive Perception & Graphics Lead, Google XR AgentHands is an LLM-powered XR prototype that augments conversational agents with synchronized, expressive hand gestures to provide spatially grounded guidance, bridging the mental mapping gap and enhancing user engagement in physical tasks. Quick links Paper As AI assistants evolve from simple text interfaces to multimodal companions, we are seeing a shift toward more proactive, situated assistance. Recent innovations like Project Astra and Gemini 3.1 Flash Live already allow users to discuss their physical surroundings in real time, often utilizing visual bounding box overlays to identify objects in a camera feed. While these overlays are highly effective for 2D screens, the transition to immersive platforms like Android XR presents a unique challenge: how do we move beyond flat UI to create a truly embodied, spatially aware dialogue? To bridge this gap, we introduce AgentHands, published at CHI 2026, a research prototype that brings the power of co-speech gestures to the 3D world. In human communication, our hands do more than just point; they describe shapes, mimic actions, and emphasize points, all synchronized with our voice. By leveraging the spatial understanding capabilities of Extended Reality (XR), AgentHands replicates this natural synergy. Following up our prior research in Human I/O and Sensible Agent, AgentHands further equips AI agents with expressive, synchronized hand gestures that transform abstract verbal instructions into intuitive, physical demonstrations, making conversations about your surroundings more natural and engaging. play silent looping video pause silent looping video unmute video mute video AgentHands demo: Empowering AI agents with expressive, synchronized hand gestures for spatially grounded conversations in XR. A taxonomy for embodied hand agents in XR To start, we conducted a formative study with XR and human–computer interaction (HCI) experts at Google to determine what makes a virtual hand “legible” in a 3D environment. We distilled these insights into a multi-dimensional taxonomy that defines how an agent should use its hands to ground a conversation within a user's physical space. Handedness & gesture:Choosing between one or two hands and selecting from a library of forms, such as a “palm” for caution or a “cylindrical grip” to mimic holding a tool. Spatiality: Leveraging the depth of XR to determine where hands should live. They can be mid-air for general conversation, object-anchored for identifying specific parts, or user-relative for social cues. Temporal dynamics & visual effects (VFX): Gestures in XR aren't just static poses; they include animated motions like “pouring” or “tracing”. We also utilize XR’s unique visual layer by adding effects, such as a red glow to signify a heat warning. The AgentHands taxonomy diagram shows the six dimensions: Handedness, Gesture, Spatiality, Temporal Dynamics, Interactivity, and Visual Effects. The AgentHands workflow The core innovation of AgentHands is its ability to map the high-level reasoning of LLMs into precise, real-time physical motions that match the agent's “voice” and the user's XR environment. We introduce the following key steps to compose the AgentHands workflow. Environment awareness The system begins with a lightweight object registration module. Using eye gaze and scene reconstruction, users can quickly “tag” items — like an orchid or a laptop — creating a spatial registry with 3D bounding boxes that the agent can reference. AgentHands uses eye gaze and scene understanding to register physical objects into a 3D registry. Hand gesture event library We constructed a library of hand gesture behaviors across three semantic categories: a) deictic for referencing, b) iconic for depicting actions or forms, and c) expression for conveying social cues and emotion. A representative set of behaviors grouped by semantic purpose to facilitate LLM selection. Gesture-embedded reasoning When a user asks a question, the backend LLM generates a response that includes inline GestureEvents. Each event is attached to specific trigger words and encodes the primitives for a hand behavior following the taxonomy dimensions. Synchronized XR execution A local parser on the XR headset coordinates the text-to-speech (TTS) playback with the animation engine. By using word-level timestamps, the agent’s hands perform co-speech gestures in perfect sync with the spoken words, providing clear, expressive spatial references. By integrating these modules, AgentHands creates a seamless bridge between linguistic intent and physical action. The system transforms a standard LLM output into a rich, multimodal performance where the agent's generated responses are manifested through both speech and spatially accurate movement, allowing for complex instructions to be demonstrated exactly where they occur in the user's environment. The full AgentHands system workflow, illustrating the flow from user speech and first-person view (FPV) to LLM-generated GestureEvents and synchronized XR rendering. Application scenarios We demonstrated how these embodied gestures, paired with the spatial awareness of XR, enhance our understanding of our physical surroundings. Interactive tutoring: In an orchid-care scenario, the agent doesn’t just say “check the roots”; it moves its hands to the base of the plant and outlines the air roots while explaining their function. play silent looping video pause silent looping video unmute video mute video AgentHands uses spatial gestures to point out aerial roots in an orchid-care task. Technical walkthroughs: For 3D printer operations, the agent can demonstrate the exact ''turn and click'' sequence needed to navigate control knobs and select files, making complex physical interface steps intuitive. play silent looping video pause silent looping video unmute video mute video AgentHands demonstrates precise knob interactions for 3D printer menu navigation. Lifestyle companionship: The agent can serve as a wellness coach that interacts with your physical choices. For instance, the agent can perform an interactive “warning” gesture by holding the user’s hand and a visual effect to caution the user against unhealthy behavior. play silent looping video pause silent looping video unmute video mute video AgentHands enacts subtle interactive gestures to support healthy daily habits. User study To evaluate the impact of these gestures, we conducted a within-subjects study (N = 12) comparing AgentHands to a speech-only baseline. Both conditions used the same researcher-scripted verbal content, ensuring the only difference was the presence of the embodied hands and their synchronized gestures. Participants completed two procedural tasks that balanced everyday care with technical operation. Orchid care task: Users learned to identify plant parts (air roots, stems) and performed multi-step care activities, including targeted watering and proper fertilizer application. 3D printer operation task: Users were guided through identifying hardware components like the nozzle and print bed, followed by a workflow to power on the device, insert an SD card, and navigate the control panel. User study setup showing the two task environments: (a) the orchid plant care station and (b) the 3D printer operation station. Results The results confirmed that the combination of XR and co-speech gestures is highly effective for spatially grounded interactions. We analyzed the data across several key metrics of communication effectiveness. Significant gains in spatial grounding: Participants found it significantly easier to locate specific objects and identify directions referred to by the agent (p < 0.05). The timely “landing” of a pointing gesture exactly as the agent said “this” eliminated the guesswork common in purely verbal instructions. Enhanced understanding of complex actions: Required activities were rated as easier to follow (p < 0.05). One participant noted that “only when the agent showed the lifting gesture did I realize I needed to lift the orchid to let the water drain”. Salient safety cues: Warnings were much more effective when paired with gestural cues and visual effects. The “burn” effect used during the 3D printer task was cited as “really impressive”, ensuring users definitely noticed the risk of a hot nozzle. Reduced cognitive load: Participants found the instructions easier to understand and remember. Qualitative feedback suggested the hands felt like a “partner guiding me”, shifting the experience from a tool-based search into a blended, embodied dialogue. Statistical results of the system feature experience questionnaire showing AgentHands significantly outperforming the baseline in location, action understanding, and warning noticeability. Conclusion and future directions AgentHands represents a step toward a future where AI systems aren’t just analyzing our world, but dynamically operating within it. By leveraging co-speech gestures and the spatial power of XR to ground conversation in physical movement, we can reduce the cognitive load of complex tasks and make spatial computing more accessible and human-centric. As we continue to develop for the Android XR ecosystem, we are exploring ways to make these gestures even more personalized, adapting to a user’s dominant hand or learning their specific spatial routines, to create an even more seamless human-AI collaboration. Acknowledgements This research was primarily conducted by Ziyi Liu during his Student Researcher tenure at Google, as part of a joint collaboration across multiple teams. We extend our sincere gratitude to key contributors David Li, Zhongyi Zhou, and David Kim for their support, and to Adarsh Kowdle, Guru Somadder, and Shahram Izadi for their strategic guidance and thoughtful reviews. Labels: Human-Computer Interaction and Visualization Machine Intelligence Quick links Paper