2026-06-17原文2 min readUpdated: 2026-06-17

Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins

We propose OR3, a method that converts OR video clips into action-driven digital twins (ActDT) and uses an LLM to generate hypothetical ActDTs from queries for imagination-based retrieval, achieving 57.6% R@1 and 77.3% R@5 on a benchmark of 276 implicit queries.

SourcearXiv Computer VisionAuthor: Yiqing Shen, Hao Ding, Mathias Unberath

[2606.17298] Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins

[Submitted on 15 Jun 2026]

Title:Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins

View a PDF of the paper titled Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins, by Yiqing Shen and 2 other authors

View PDF HTML (experimental)

Abstract:Text-to-video retrieval in operating rooms (OR) is an enabling technology for OR safety, as it allows stakeholders to retrieve and inspect recordings of specific events. However, because the most safety-critical events may not follow the common structure, to unlock its full potential text-to-video retrieval must be able to handle implicit queries that require reasoning to identify the right video (e.g., the step right before clipping). However, existing methods rely on global embeddings that cannot reason over such queries. We propose OR3, a text-to-video retrieval method that converts clips into action-driven digital twins (ActDTs), grouping concurrent subject-action-object triplets under non-overlapping temporal intervals. Moreover, rather than cross-modal matching through paired encoders, OR3 performs imagination-based retrieval where an LLM generates hypothetical ActDTs from queries. This enables intra-modal matching via a single encoder trained with ActDT-tailored hard negatives. Finally, evidence-grounded refinement revises imagined ActDTs based on discrepancies with top candidates to capture procedure-specific patterns. We construct a benchmark from MM-OR with 276 implicit queries across four reasoning categories over 386 clips from robotic knee procedures. OR3 achieves 57.6 R@1 and 77.3 R@5, outperforming the strongest baseline. These results demonstrate that OR3 enables fine-grained discrimination between visually similar OR video clips through temporal action reasoning.

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2606.17298 [cs.CV]

(or arXiv:2606.17298v1 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2606.17298

arXiv-issued DOI via DataCite

Submission history

From: Yiqing Shen [view email] [v1] Mon, 15 Jun 2026 21:06:53 UTC (657 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins, by Yiqing Shen and 2 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CV

new | recent | 2026-06

Change to browse by:

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)