JavaScript library for rapid AI and XR prototyping

Uh oh!

There was an error while loading. Please reload this page.

Notifications You must be signed in to change notification settings

Fork 83

Star 429

BranchesTags

Open more actions menu

Folders and files

NameName

Last commit message

Last commit date

Latest commit

History

1,363 Commits

1,363 Commits

.github

.github

.vscode

.vscode

assets

assets

demos

demos

docs

docs

evals

evals

samples

samples

skills

skills

src

src

templates

templates

.gitignore

.gitignore

.prettierignore

.prettierignore

.prettierrc.json

.prettierrc.json

AGENTS.md

AGENTS.md

CITATION.cff

CITATION.cff

CLAUDE.md

CLAUDE.md

CONTEXT.md

CONTEXT.md

CONTRIBUTING.md

CONTRIBUTING.md

GEMINI.md

GEMINI.md

LICENSE

LICENSE

README.md

README.md

eslint.config.mjs

eslint.config.mjs

package-lock.json

package-lock.json

package.json

package.json

rollup.config.js

rollup.config.js

tsconfig.base.json

tsconfig.base.json

tsconfig.json

tsconfig.json

vitest.config.ts

vitest.config.ts

Repository files navigation

JavaScript library for rapid AI + XR prototyping

Site — Manual — Templates — Demos — Gem — YouTube — arXiv — Blog

Description

XR Blocks is a lightweight, cross-platform library for rapidly prototyping advanced XR and AI experiences. Built upon three.js, it targets Chrome v136+ with WebXR support on Android XR (e.g., Galaxy XR) and also includes a powerful desktop simulator for development. The framework emphasizes a user-centric, developer-friendly SDK designed to simplify the creation of immersive applications with features like:

Hand Tracking & Gestures: Access advanced hand tracking, custom gestures with TensorFlow Lite / PyTorch models, and interaction events.

Gesture Recognition: Opt into pinch, open-palm, fist, thumbs-up, point, and spread detection with options.enableGestures(), tune providers or thresholds, and subscribe to gesturestart/gestureupdate/gestureend events from the shared subsystem.

World Understanding: Present samples with depth sensing, geometry-aware physics, and object recognition with Gemini in both XR and desktop simulator.

AI Integration: Seamlessly connect to Gemini for multimodal understanding and live conversational experiences.

Cross-Platform: Write once and deploy to both XR devices and desktop Chrome browsers.

Automation-Friendly Simulator: Use options.enableAutomationMode() or ?xrAutomation=1 to start a desktop simulator preset for external remote runs.

We welcome all contributors to foster an AI + XR community! Read our blog post and white paper for a visionary roadmap.

Usage

XR Blocks can be imported directly into a webpage using an importmap. This code creates a basic XR scene containing a cylinder. When you view the scene, you can pinch your fingers (in XR) or click (in the desktop simulator) to change the cylinder's color. Check out this live demo with simple code below:

Basic Example | XR Blocks

Gem

For rapid prototyping, please refer to this blog and try out XR Blocks Gem on Android XR or desktop using Chrome.

A video demonstration of XR Blocks Gem can be found here.

Development Guide

Setup

Clone the repository.

git clone --depth=1 [email protected]:google/xrblocks.git cd xrblocks

Install dependencies and build the SDK.

npm ci

After setup, either serve the samples and demos or develop locally, as described below.

Serve samples and demos

Serve the repository to view samples and demos through http://localhost:8080/:

Serve the repository on http://localhost:8080

npm run serve

Develop locally

For active SDK development, run watch mode and local serving together:

Build the SDK in watch mode and serve the repository on http://localhost:8080

npm run dev

Linting and formatting

XR Blocks uses ESLint for linting and Prettier for formatting. Run the following commands to check your code before submitting a pull request:

npm run lint # ESLint check npm run format # Prettier format

If you are using VS Code, install the ESLint extension and the Prettier extension, then set Prettier as your default formatter.

Notice

This is not an officially supported Google product, but will be actively maintained by the XR Labs team and external collaborators. This project is not eligible for the Google Open Source Software Vulnerability Rewards Program.

User Data & Permissions

When using specific features in this SDK (e.g., WebXR, hand tracking, camera), users will be prompted with permission requests and the application may not function as expected with denied permissions.

XR Blocks is an open source software development kit that does not handle data by itself; however, the use of other APIs may collect user data and require user permissions:

When using WebXR and LiteRT APIs (e.g., depth sensing, gesture recognition), all data is stored and processed locally with on-device models.

When using AI features (e.g., Gemini Live, Gemini Flash), the data will be sent to Gemini servers and please follow Gemini's Privacy & Terms.

Keep Your API Key Secure

This SDK does not require any API keys for non-AI samples. In specific AI use cases, this SDK provides an interface to use cloud-hosted Gemini services with XR experiences, requiring an API key from AI Studio. Please follow this doc for best practices to keep your API key secure.

Treat your Gemini API key like a password. If compromised, others can use your project's quota, incur charges (if billing is enabled), and access your private data, such as files.

Critical Security Rules

Never commit API keys to source control. Do not check your API key into version control systems like Git.

Never expose API keys on the client-side. Do not use your API key directly in web or mobile apps in production. Keys in client-side code (including our JavaScript/TypeScript libraries and REST calls) can be extracted.

Uninstallation

To remove XR Blocks from your code, simply remove the lines from your tag in HTML, or import * from xrblocks in JavaScript, or use npm uninstall xrblocks from your project directory.

References

If you find XR Blocks inspiring or useful in your research, please reference it as:

@misc{Li2025XR, title={{XR Blocks: Accelerating Human-centered AI + XR Innovation}}, author={Li, David and Numan, Nels and Qian, Xun and Chen, Yanhe and Zhou, Zhongyi and Alekseev, Evgenii and Lee, Geonsun and Cooper, Alex and Xia, Min and Chung, Scott and Nelson, Jeremy and Yuan, Xiuxiu and Dias, Jolica and Bettridge, Tim and Hersh, Benjamin and Huynh, Michelle and Piascik, Konrad and Cabello, Ricardo and Kim, David and Du, Ruofei}, year={2025}, eprint={2509.25504}, archivePrefix={arXiv}, primaryClass={cs.HC}, url={https://arxiv.org/abs/2509.25504}, }

@misc{Du2026Vibe, title={Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini}, author={Ruofei Du and Benjamin Hersh and David Li and Nels Numan and Xun Qian and Yanhe Chen and Zhongyi Zhou and Xingyue Chen and Jiahao Ren and Robert Timothy Bettridge and Steve Toh and David Kim}, year={2026}, eprint={2603.24591}, archivePrefix={arXiv}, primaryClass={cs.HC}, url={https://arxiv.org/abs/2603.24591}, }

Terms of Service

Please follow Google's Privacy & Terms when using this SDK.

When using AI features in this SDK, please follow Gemini's Privacy & Terms.

About

XR Blocks is a lightweight WebXR + AI library for rapidly prototyping advanced AI + XR experiences.

xrblocks.github.io

Topics

tensorflow

gemini

gestures

webxr

litert

webai

androidxr

xrblocks

Resources

Readme

License

Apache-2.0 license

Code of conduct

Code of conduct

Contributing

Contributing

Security policy

Security policy

Uh oh!

There was an error while loading. Please reload this page.

Activity

Custom properties

Stars

429 stars

Watchers

4 watching

Forks

83 forks

Report repository

Releases 22

v0.17.0: Remote/Embodied Control, Headless Testing & Advanced Perception

Latest

Jun 30, 2026

+ 21 releases

Packages 0

Uh oh!

There was an error while loading. Please reload this page.

Uh oh!

There was an error while loading. Please reload this page.

Contributors

Uh oh!

There was an error while loading. Please reload this page.

Languages

TypeScript 92.2%

HTML 3.3%

JavaScript 2.4%

Python 2.0%

Shell 0.1%