Coding in space, AI-XR, and new interaction paradigms for devs

JetBrains Research

Research is crucial for progress and innovation, which is why at JetBrains we are passionate about both scientific and market research

About JetBrains Research

Research

Code in Space: Redefining Tech Creation with AI and XR

The main message

Advances in AI are already single-handedly changing how we interact with technologies. This moment might become the first interaction revolution in roughly 60 years, since the mouse-and-windowing paradigm crystallized in the 1970s and 1980s. For us at JetBrains, the most interesting part of this interaction is related to code. LLMs have introduced the conversational format to software engineering. Something that might appear no less consequential than AI several years from now is extended reality (XR) hardware, and how it is (slowly) becoming good enough to be taken seriously as a working medium. XR is relevant for AI not merely as a more immersive output device; it is, even more importantly, an unprecedentedly rich input surface: using gaze, hands, voice, head pose, body posture, spatial context, and even physiological signals. Altogether, with all these new types of data, it enables a more efficient, focused, and personalized multimodal human-AI experience. Crucially, all these inputs are available without a dedicated lab.

When the input richness of XR is paired with modern AI, the result is a qualitatively new interaction substrate, one we’re just starting to figure out how to use. Our Human-AI eXperience (HAX) team at JetBrains Research is now studying what this means specifically for tech creators. By tech creators, we mean software engineers, of course, but also hardware engineers, UI/UX designers, and data scientists, as the question is broader than any one role.

What we asked

To anchor our exploration of what XR can bring to HAX empirically, we conducted semi-structured expert interviews. Thirteen senior researchers and practitioners from leading academic institutions and industry labs (including groups at Cambridge, Aarhus, Stuttgart, and Meta) participated. We analyzed these interviews using the classic Braun and Clarke’s (2006) six-phase approach. The interviews continued until we reached thematic saturation: when three consecutive interviews introduced no new parent themes.

We were interested in understanding:

1.What are the needs and recurring pain points of tech creators that multimodal AI-XR tools could plausibly address?

2.Which multimodal interaction techniques are most promising for addressing those needs, and what might the resulting tooling look like?

3.What are the main constraints – technical, cognitive, organizational – for building such tools today?

What we found, in brief

We’ve identified more than 150 topics of interest from the answers to our interview questions. All these topics were then clustered (first independently by two of the authors and then updated until a consensus was reached) into five overarching themes.

AI-XR consolidated codebook — tabbed

The initial codes and thematic clusters of AI-XR for multimodal Human-AI Experience

156 consolidated codes across 5 themes

How do humans communicate intent to AI-XR systems?

New interaction paradigms for spatial computing Future of interactions Natural interaction Multimodal context and input No preferences in multimodality New inputs and UI placement Matching input with the task Explicit and implicit interactions AI-based hand and gesture tracking AI-based voice control AI-based body language AI-based haptics and tactile feedback Gaze control No controllers for XR Embodied interaction Personalization of inputs Spatial reasoning Wearables for XR AI-XR communication and telepresence Switching perspectives for communication Human communication-inspired interaction Better UI Visual overlaying Mobile/on-the-go interaction Flexible interaction patterns Feeling depth in XR Audio is underused Vocalization will not be enough Lack of freedom for interaction Gorilla arm / interaction fatigue Midas touch problem “No interface” / zero-UI paradigm Constraints can aid efficiency 2D metaphors are poor fits for 3D

Can AI make XR environments understand and adapt to you?

AI as a context-aware co-creator and mediator Agentic AI AI for understanding context/user Inherent personalization of AI Personalized XR through AI AI lacking personal/user context AI for content creation Communicating/decoding intent AI for disambiguation Immediate interactive context/content Interactive environment Directing attention Attention as a scarce resource Digital twinning Semantic graphs from visual scenes Environmental sensing and physical-world understanding Multimodal data collection New data for learning in XR AI for tracking body and behavior Humanlike behavior of AI in 3D Control of environment AI-XR guidance and explanation XR for navigation Always-on/ambient AI-XR Continuous AI-XR experience Proactive/predictive AI-XR Ubiquitous/situated computing in XR Situated/physical-world computing and augmentation Diminished reality AI-XR augmenting people Merging physical spaces Rule-based systems and predetermined patterns limit AI Incomplete AI models of digital/physical context

What is actually blocking XR-AI from becoming mainstream?

Feasibility, adoption, and ecosystem readiness Technological and hardware barriers Technological availability Computational requirements Insufficient bandwidth Cost barriers New technical challenges New infrastructure for AI Interaction precision and resolution limits Calibration vs. quality trade-off (eye tracking) Responsiveness of input and system Trade-offs of AI-XR XR proving its value XR not improving AI Barriers in personal performance and user experience Ergonomics Intrusive wearables Aesthetics of XR Social acceptability XR for public settings XR UI for long sessions Frustrating experience Initial friction of adoption Easy step-in/step-out from XR workflows Fast restart Workflow transition/switching friction AI-assisted workflow transitions 3D metaphors 3D is not always better than 2D Text is bad outside desktop Text-centered ecosystem XR UI for tech is the same as general XR UI Lack of specificity of AI for technical work SE in XR is niche SE is more demanding for interaction Lack of human connection Interoperability and shared XR standards Content creation intertwined with coding

How does AI-XR change how people make things?

XR as a situated, persistent workspace Extended workspace for coding and engineering Mobile workstations XR as primary computing/display platform Personalized organization of the XR workspace XR/AI-XR visualization for data and complex systems XR for robotics/teleoperation Teaching and understanding 3D XR for teaching and training AI-XR prototyping and 3D prototyping AI-XR sketching, drawing, and design AI for general productivity AI for deep research AI as counterfactual / devil’s advocate AI-XR collaboration XR for human-AI interaction (HAX) AI-XR multi-tasking XR avatars and digital self-representation AI-XR video creation/editing End-to-end creation in XR Iterative creation in XR AI-XR for creating variations Tools for 3D engineering Tools for technology creation Tech creation when no more applications Writing code is not the core concern anymore VR for 3D printing XR for game development/gaming 4D videography Immersive tourism Deep immersion

As AI-XR systems grow more capable and ambient, what do humans lose or risk?

Privacy concerns / convenience trade-off Security in XR (next-level phishing) Ethical concerns Governmental regulations AI regulations do not work in advance Rights, authorship, and ownership AI transparency and black-box concerns XR for explaining AI and transparency Alignment Preserving human-in-the-loop Technology overreliance Overuse of AI features (e.g., beautify) Potential exploitation through XR Commercialization of AR/XR environments Societal changes Social concerns Changing emotional landscape 24/7 technology use Constantly-on devices Digital copies, twinning, and identity risks

AI-XR for creation and professional work. This is the cluster most directly related to our main question: What is the future of technology creation? Here the main questions center on the productive use: an XR environment as a place where knowledge workers, designers, and engineers actually build things. The emerging interaction insight is that the boundary between authoring, coding, and designing is dissolving in XR, and AI is accelerating that dissolution. One unexpected topic that stood out here and genuinely surprised us was framing the XR for robotics as a potential sandbox for training the next generation of machine learning models. The idea is the following: if you put simulated robots into a highly controlled virtual environment with rich structured multimodal data, and then create an interaction loop within this virtual environment, you can then use that loop to generate the kind of physical interaction data that today’s language and vision models mostly lack.

AI’s impact on coding is, at this point, well-documented and widely discussed. Together with other researchers, our group is trying to figure out these changes through surveys, longitudinal studies, and analyses of developers’ evolving needs. However, we believe another big change is coming. Consider the output we already see: Consumer-grade XR hardware is in the middle of a rapid generational shift.

The Apple Vision Pro is currently the most visible example of this push toward high-resolution spatial computing (although it’s now on pause, it’s still a big thing for the industry). However, other devices like PICO or the ultra-lightweight BigScreen Beyond are proving that form factors and devices are diversifying rapidly. Google is also pushing boundaries in this space, notably with their work on vibe-coding XR using XR Blocks and Gemini. While the ecosystem around these gadgets is still sparse, it is growing very quickly. None of these are perfect, and none are a complete replacement for a workstation. The trajectory is unmistakable, however: headsets are getting lighter, displays are sharper, tracking is more reliable, and ecosystems are thicker.

In parallel (and most easily overlooked), the input side is exploding. Eye tracking is becoming a default rather than an accessory. Hand tracking has reached the precision needed for controller-free interaction to be a viable tool for many tasks. Beyond that, a generation of new sensors is moving from research lab prototypes toward future integration: ear-EEG that captures brain activity from the auditory canal, electromyography-based silent speech interfaces, fine-grained physiological monitoring (e.g. heart rate variability, pupillometry, galvanic skin response), and high-resolution facial and posture tracking. The image below shows examples of new sensor types from Tang et al (2026).

Lab sensor types for new inputs grouped by proximity to the body: off-body (e.g. cameras, smart glasses), on-body (e.g. EEG hats, patches), and in-body implants. These categories highlight a continuous trade-off between user comfort and signal accuracy, ranging from general-purpose wearables to highly precise, clinically validated medical systems.

Take capable headsets and an unprecedentedly broad input bandwidth together, add modern multimodal AI, and you arrive at something genuinely new. This is the territory our project is trying to map.

To calibrate our research against the state of the field, a researcher from our team attended IEEE VR 2026 in Daegu, Korea, one of the leading conferences covering virtual, augmented, and mixed reality. The picture that emerged was useful both for what it confirmed and for what it revealed to be complicated.

The picture confirmed that XR is no longer primarily a graphics or hardware community. The program — paper tracks across three days and four parallel sessions — covered haptic feedback and rendering, multimod

[truncated for AI cost control]