AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。
In this tutorial, we work through the retargeting engine at the core of NVIDIA IsaacTeleop, the framework that turns XR hand tracking and motion-controller input into commands for simulated and real robots. Rather than plugging in a headset, we build every input ourselves in NumPy, so each step runs on a plain Colab CPU and prints what it computes. We start with the type system every node speaks, generate synthetic hand and controller data, write our own retargeter with live-tunable parameters, and then drive the built-in gripper and SE(3) retargeters with it. From there, we compose a full graph that emits one action vector per step, apply a world-frame transform, step through the run, pause and kill the state machine, and finish with a controller-to-dexterous-hand mapping and parameter tuning that persists across restarts. Copy CodeCopiedUse a different Browser import os import sys import json import math import tempfile import traceback import subprocess import numpy as np RESULTS = {} def banner(title): print("\n" + "=" * 78) print(title) print("=" * 78) def section(name): def wrap(fn): def run(*a, kw): banner(name) try: out = fn(*a, kw) RESULTS[name] = out if isinstance(out, str) else "ok" return out except Exception as e: RESULTS[name] = f"SKIPPED / FAILED -> {type(e).name}: {e}" print(f"\n[!] {name} did not complete: {type(e).name}: {e}") traceback.print_exc(limit=3) return None return run return wrap banner("0. Install isaacteleop and check the environment") subprocess.run( [sys.executable, "-m", "pip", "install", "-q", "isaacteleop[retargeters-lite]==1.4.145"], check=True, ) import pkgutil import isaacteleop from isaacteleop import schema print(f" isaacteleop {isaacteleop.version} | Python {sys.version.split()[0]} | numpy {np.version}") print(" top-level modules :", ", ".join(sorted(m.name for m in pkgutil.iter_modules(isaacteleop.path)))) message_types = [n for n in dir(schema) if n[0].isupper()] print(f" schema message types: {len(message_types)}, e.g. {', '.join(message_types[:6])}") print(" no headset, no OpenXR runtime, no simulator is used anywhere below.") We install the stable isaacteleop wheel from PyPI with the retargeters-lite extra, which adds only SciPy, and report the version, interpreter, and NumPy build. The package splits into device I/O modules that wrap OpenXR and CloudXR, a schema module holding the FlatBuffer message types every tracker emits, and the pure-Python retargeting engine we work with here. Listing the schema types shows the vocabulary of the data layer, but nothing below opens a headset session; from here on, every input is a tensor we build by hand. Copy CodeCopiedUse a different Browser from isaacteleop.retargeting_engine.interface import ( TensorGroup, OptionalTensorGroup, TensorGroupType, OptionalType, ) from isaacteleop.retargeting_engine.tensor_types import ( HandInput, ControllerInput, HandInputIndex, ControllerInputIndex, HandJointIndex, ) @section("1. The type contract: TensorGroupType, TensorGroup, Optional") def type_contract(): hand_t = HandInput() ctrl_t = ControllerInput() print(" HandInput :", hand_t) print(" slot index :", ", ".join(f"{m.name}={m.value}" for m in HandInputIndex)) print(f" ControllerInput: {len(ctrl_t)} slots ->", ", ".join(t.name.replace("controller_", "") for t in ctrl_t.types)) print(f" OpenXR hand joints: {len(HandJointIndex)} " f"(WRIST={int(HandJointIndex.WRIST)}, THUMB_TIP={int(HandJointIndex.THUMB_TIP)}, " f"INDEX_TIP={int(HandJointIndex.INDEX_TIP)})") hand = TensorGroup(hand_t) hand[HandInputIndex.JOINT_POSITIONS] = np.zeros((26, 3), dtype=np.float32) print("\n wrote a (26, 3) float32 array into JOINT_POSITIONS ->", hand) try: hand[HandInputIndex.JOINT_POSITIONS] = np.zeros((26, 3)) except TypeError as e: print(" float64 rejected at write time :", str(e)[:110]) try: _ = hand[HandInputIndex.JOINT_VALID] except ValueError as e: print(" reading a slot nobody wrote :", e) maybe = OptionalTensorGroup(OptionalType(ctrl_t)) print(f"\n Optional group starts absent : is_none={maybe.is_none} {maybe}") maybe[ControllerInputIndex.TRIGGER_VALUE] = 0.0 print(f" one write flips it to present : is_none={maybe.is_none} {maybe}") return f"{len(hand_t)} hand slots, {len(ctrl_t)} controller slots" type_contract() The engine’s contract is a TensorGroupType, an ordered list of typed slots, and a TensorGroup, the runtime container that holds one value per slot and validates each write. HandInput carries four NumPy arrays for the 26 OpenXR hand joints, and ControllerInput carries fourteen slots for poses, buttons, and axes, addressed through generated IntEnum indices rather than magic numbers. Writing a float64 array where float32 is declared fails at the write, and reading a slot nobody wrote raises instead of returning stale data. OptionalType marks inputs a tracker may not deliver; the matching OptionalTensorGroup starts absent and becomes present on its first write, which is how every downstream node learns that a hand has left the tracking volume. Copy CodeCopiedUse a different Browser J = HandJointIndex C = ControllerInputIndex RIGHT_WRIST = np.array([0.30, 1.05, -0.45], dtype=np.float32) # OpenXR: x right, y up, z back def make_hand(pinch_m, wrist=RIGHT_WRIST, quat=(0.0, 0.0, 0.0, 1.0)): """A synthetic right hand in HandInput layout: 26 joints in OpenXR order.""" pos = np.zeros((26, 3), dtype=np.float32) pos[J.WRIST] = wrist pos[J.PALM] = wrist + [0.0, 0.0, -0.06] fingers = [(J.INDEX_METACARPAL, 0.03), (J.MIDDLE_METACARPAL, 0.01), (J.RING_METACARPAL, -0.01), (J.LITTLE_METACARPAL, -0.03)] for base, x_off in fingers: # metacarpal .. tip, five joints each for k in range(5): pos[base + k] = wrist + [x_off, 0.0, -0.05 - 0.02 * k] for k in range(4): # thumb: metacarpal .. tip, four joints pos[J.THUMB_METACARPAL + k] = wrist + [0.05, -0.01, -0.02 - 0.015 * k] pos[J.THUMB_TIP] = pos[J.INDEX_TIP] + [pinch_m, 0.0, 0.0] g = TensorGroup(HandInput()) g[HandInputIndex.JOINT_POSITIONS] = pos g[HandInputIndex.JOINT_ORIENTATIONS] = np.tile(np.asarray(quat, dtype=np.float32), (26, 1)) g[HandInputIndex.JOINT_RADII] = np.full(26, 0.01, dtype=np.float32) g[HandInputIndex.JOINT_VALID] = np.ones(26, dtype=np.uint8) return g def make_controller(pos, quat=(0.0, 0.0, 0.0, 1.0), trigger=0.0, squeeze=0.0, thumbstick=(0.0, 0.0), valid=True): """A synthetic controller snapshot in ControllerInput layout.""" p = np.asarray(pos, dtype=np.float32) q = np.asarray(quat, dtype=np.float32) g = TensorGroup(ControllerInput()) g[C.GRIP_POSITION], g[C.GRIP_ORIENTATION], g[C.GRIP_IS_VALID] = p, q, bool(valid) g[C.AIM_POSITION] = p + np.array([0.0, 0.0, -0.05], dtype=np.float32) g[C.AIM_ORIENTATION], g[C.AIM_IS_VALID] = q.copy(), bool(valid) for idx in (C.PRIMARY_CLICK, C.SECONDARY_CLICK, C.THUMBSTICK_CLICK, C.MENU_CLICK): g[idx] = 0.0 g[C.THUMBSTICK_X], g[C.THUMBSTICK_Y] = float(thumbstick[0]), float(thumbstick[1]) g[C.SQUEEZE_VALUE], g[C.TRIGGER_VALUE] = float(squeeze), float(trigger) return g @section("2. Synthetic tracking data: a hand and a controller in numpy") def synthetic_inputs(): hand = make_hand(pinch_m=0.06) pos = hand[HandInputIndex.JOINT_POSITIONS] print(" joint x y z") for j in (J.WRIST, J.PALM, J.THUMB_TIP, J.INDEX_TIP, J.LITTLE_TIP): print(f" {j.name:12s} {pos[j][0]:6.3f} {pos[j][1]:6.3f} {pos[j][2]:6.3f}") pinch = np.linalg.norm(pos[J.THUMB_TIP] - pos[J.INDEX_TIP]) print(f" thumb-to-index distance: {pinch * 100:.1f} cm " f"valid joints: {int(hand[HandInputIndex.JOINT_VALID].sum())}/26") ctrl = make_controller((0.40, 1.20, -0.30), trigger=0.8, thumbstick=(0.0, 0.5)) print(f"\n controller grip {np.round(ctrl[C.GRIP_POSITION], 2)} aim {np.round(ctrl[C.AIM_POSITION], 2)} " f"trigger {ctrl[C.TRIGGER_VALUE]} squeeze {ctrl[C.SQUEEZE_VALUE]} " f"thumbstick ({ctrl[C.THUMBSTICK_X]}, {ctrl[C.THUMBSTICK_Y]})") return "synthetic HandInput + ControllerInput builders" synthetic_inputs() We build the tracking data that a headset would normally supply. make_hand lays out 26 joints in OpenXR order around a wrist position, runs four finger chains and a thumb chain away from the palm, and places the thumb tip a chosen pinch distance from the index tip, so the one number the later steps depend on is under our control. make_controller fills every ControllerInput slot: grip and aim poses with validity flags, four buttons, the thumbstick axes, and the analog squeeze and trigger values. Both return ordinary TensorGroups, which is all a retargeter ever sees, whether the numbers came from OpenXR or from NumPy. Copy CodeCopiedUse a different Browser from isaacteleop.retargeting_engine.interface import ( BaseRetargeter, ParameterState, FloatParameter, BoolParameter, ) from isaacteleop.retargeting_engine.tensor_types import FloatType, BoolType class PinchRetargeter(BaseRetargeter): """Thumb-to-index distance -> (distance_cm, is_pinching), with live-tunable parameters.""" def init(self, name, config_file=None): params = [ FloatParameter("threshold_cm", "Pinching when closer than this", default_value=3.0, min_value=0.5, max_value=10.0, sync_fn=lambda v: setattr(self, "threshold_cm", v)), BoolParameter("use_distal_joints", "Measure between distal joints, not tips", default_value=False, sync_fn=lambda v: setattr(self, "use_distal_joints", v)), ] super().init(name, parameter_state=ParameterState(name, params, config_file=config_file)) def input_spec(self): return {"hand_right": OptionalType(HandInput())} def output_spec(self): return {"pinch": TensorGroupType("pinch", [FloatType("distance_cm"), BoolType("is_pinching")])} def _compute_fn(self, inputs, outputs, context): hand = inputs["hand_right"] if hand.is_none: # tracking lost: say so, do not guess outputs["pinch"][0] = -1.0 outputs["pinch"][1] = False return pos = np.from_dlpack(hand[HandInputIndex.JOINT_POSITIONS]) a, b = (J.THUMB_DISTAL, J.INDEX_DISTAL) if self.use_distal_joints else (J.THUMB_TIP, J.INDEX_TIP) d_cm = float(np.linalg.norm(pos[a] - pos[b]) * 100.0) outputs["pinch"][0] = d_cm outputs["pinch"][1] = bool(d_cm distance_cm={out['pinch'][0]:.2f} is_pinching={out['pinch'][1]}") out = pinch({}) # optional input omitted entirely print(f" no hand tracked -> distance_cm={out['pinch'][0]:.1f} is_pinching={out['pinch'][1]}") state = pinch.get_parameter_state() print("\n tunable parameters:", state.get_all_values()) for threshold in (2.5, 3.5): state.set({"threshold_cm": threshold}) # what the tuning UI does from its own thread out = pinch({"hand_right": make_hand(0.028)}) print(f" threshold_cm={threshold}: a 2.8 cm pinch -> is_pinching={out['pinch'][1]}") state.set({"use_distal_joints": True}) out = pinch({"hand_right": make_hand(0.028)}) print(f" use_distal_joints=True: distance_cm={out['pinch'][0]:.2f} is_pinching={out['pinch'][1]}") return "custom retargeter + 2 tunable parameters" custom_retargeter() A retargeter is a BaseRetargeter subclass that declares input_spec and output_spec and implements _compute_fn; the framework fills missing optional inputs with absent groups, validates types, syncs parameters, and only then calls our code. PinchRetargeter measures the thumb-to-index distance and emits a float and a bool, reporting -1 when no hand is tracked instead of guessing. Its two parameters live in a ParameterState whose sync functions write onto the instance before every compute, so a value set from another thread by the tuning UI takes effect on the next frame. We reproduce that by calling set directly: the same 2.8 cm pinch flips between not pinching and pinching as the threshold moves, and switching the measurement to the distal joints changes the distance itself. Copy CodeCopiedUse a different Browser from isaacteleop.retargeters import ( GripperRetargeter, GripperRetargeterConfig, Se3AbsRetargeter, Se3RelRetargeter, Se3RetargeterConfig, ) @section("4. Built-in retargeters: gripper hysteresis and SE(3) end-effector pose") def builtin_retargeters(): gripper = GripperRetargeter( GripperRetargeterConf [truncated for AI cost control]