Skip to content
AI News HubLIVE
Original source2 min read

IntentTwin: Deconfounding Candidate Support and the Oracle-to-Deployment Gap in Offline Web Behavior Prediction

Summary

This paper reveals a candidate-support confound in offline web-behavior benchmarks where inserting the demonstrated action into a finite candidate set allows support membership alone to identify the gold action. It introduces support-identifiability rate (SIR) to quantify this effect, constructs support-balanced candidate sets, and evaluates a target-free dual-path proposer on OPeRA and Mind2Web, showing that controlling support composition is essential when using offline results to reason about deployment.

SourcearXiv Computational LinguisticsAuthor: Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Daren Zha
IntentTwin: Deconfounding Candidate Support and the Oracle-to-Deployment Gap in Offline Web Behavior Prediction
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[Submitted on 27 Oct 2026]

Title:IntentTwin: Deconfounding Candidate Support and the Oracle-to-Deployment Gap in Offline Web Behavior Prediction

View a PDF of the paper titled IntentTwin: Deconfounding Candidate Support and the Oracle-to-Deployment Gap in Offline Web Behavior Prediction, by Miaobo Hu and 5 other authors

View PDF HTML (experimental)

Abstract:Offline web-behavior benchmarks frequently insert the demonstrated action into a finite candidate set and then measure reranking. We show that this protocol can create a candidate-support confound: when the target is outside training support while all distractors are sampled from that support, support membership identifies the gold action without modeling behavior. We formalize this effect with support-identifiability rate (SIR), construct support-balanced candidate sets, and evaluate a target-free dual-path proposer that combines support retrieval with open-vocabulary DOM/LM proposals. On filtered OPeRA (4,282/469/992 train/dev/test steps), the original protocol has SIR 1.0000 on target-OOV rows and a support-only rule reaches 1.0000 OOV Exact@1. Support balancing reduces SIR to 0.0000; the strongest conditional reranker obtains 0.7214 overall Exact@1, 0.7988 on IV rows, and 0.5668 on OOV rows. The target-free dual-path system reaches Proposal Recall@16 0.6915 and end-to-end Exact@1 0.5212, compared with oracle conditional Exact@1 0.7450. On Mind2Web, the same audit yields SIR 0.2380 and an oracle-to-proposal gap of 0.2125. A 200-row semantic audit labels 0.7150 of exact-key OOV targets as semantically novel, 0.2450 as locator/string variation, and 0.0400 as ambiguous (Cohen's $\kappa=0.8173$). These measurements show why support composition must be controlled alongside candidate accuracy when offline results are used to reason about deployment.

Comments: 26 pages, 10 figures

Subjects:

Computation and Language (cs.CL); Information Retrieval (cs.IR); Machine Learning (cs.LG)

Cite as: arXiv:2611.12918 [cs.CL]

(or arXiv:2611.12918v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2611.12918

arXiv-issued DOI via DataCite

Submission history

From: Bokun Wang [view email] [v1] Tue, 27 Oct 2026 18:39:23 UTC (1,841 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled IntentTwin: Deconfounding Candidate Support and the Oracle-to-Deployment Gap in Offline Web Behavior Prediction, by Miaobo Hu and 5 other authors

View PDF

HTML (experimental)

TeX Source

view license

Additional Features

Audio Summary

Current browse context:

cs.CL

new | recent | 2026-11

Change to browse by:

cs cs.IR cs.LG

References & Citations

SciX

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • Introduces support-identifiability rate (SIR) to formalize the candidate-support confound: when targets are outside training support and distractors are sampled from it, support membership alone can identify the gold action without behavior modeling.
  • On OPeRA, the original protocol yields SIR 1.0000 on target-OOV rows, with a support-only rule achieving 1.0000 OOV Exact@1; support balancing reduces SIR to 0.0000, and the strongest conditional reranker reaches 0.7214 overall Exact@1.
  • A target-free dual-path system combining support retrieval with open-vocabulary DOM/LM proposals achieves Proposal Recall@16 0.6915 and end-to-end Exact@1 0.5212 on OPeRA, versus oracle conditional Exact@1 0.7450.
  • On Mind2Web, the audit yields SIR 0.2380 and an oracle-to-proposal gap of 0.2125; a semantic audit labels 71.5% of exact-key OOV targets as semantically novel, 24.5% as locator/string variation, and 4.0% as ambiguous.

Highlights and analysis are generated automatically and may contain errors. Check the original source.

IntentTwin: Deconfounding Candidate Support and the Oracle-to-Deployment Gap in Offline Web Behavior Prediction | AI News Hub