Abstention-Aware Personalized Object Rearrangement via Uncertainty-Guided LLM Assistance
This paper introduces APOLLO, a hybrid framework that combines a lightweight personalized embedding model with selective large language model (LLM) assistance for abstention-aware object rearrangement in cluttered, partially erroneous environments. The framework uses uncertainty estimates to decide when to invoke the LLM, balancing efficiency, privacy, and reasoning. A new synthetic dataset, APOR, is also introduced. Experiments show APOLLO reduces LLM usage while maintaining or improving performance over prior methods.
[2606.17309] Abstention-Aware Personalized Object Rearrangement via Uncertainty-Guided LLM Assistance
[Submitted on 15 Jun 2026]
Title:Abstention-Aware Personalized Object Rearrangement via Uncertainty-Guided LLM Assistance
View a PDF of the paper titled Abstention-Aware Personalized Object Rearrangement via Uncertainty-Guided LLM Assistance, by Sam Collin and 1 other authors
View PDF HTML (experimental)
Abstract:Robotic assistance in household environments requires not only predicting where objects should be placed, but also reasoning about when objects should not be placed at all. Existing approaches to personalized object rearrangement primarily focus on placement decisions under the assumption of clean observations and complete actionability, limiting their applicability in realistic, cluttered, and partially erroneous settings. In this paper, we introduce APOLLO, a hybrid framework for abstention-aware personalized object rearrangement that combines a lightweight, personalized embedding model (PEM) with selective large language model (LLM) assistance. PEM is trained for each user-environment pair using a small number of demonstrations, operates entirely on CPU, and produces uncertainty estimates, which are used to selectively invoke LLM-based reasoning only for ambiguous decisions, balancing efficiency, privacy, and reasoning capability. To evaluate this formulation beyond existing benchmarks, we introduce APOR, a synthetic, LLM-generated dataset that captures room-level, multi-furniture environments, diverse organizational profiles, explicit abstention behavior, and noisy partial scene context. Extensive experiments on both PARSEC and APOR provide initial evidence that APOLLO improves over prior LLM-based baselines in controlled benchmark settings while substantially reducing LLM usage. Code is available at this https URL.
Comments: Accepted at the 2026 IEEE 35th International Conference on Robot and Human Interactive Communication (RO-MAN 2026)
Subjects:
Robotics (cs.RO)
Cite as: arXiv:2606.17309 [cs.RO]
(or arXiv:2606.17309v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2606.17309
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Sam Collin [view email] [v1] Mon, 15 Jun 2026 21:26:48 UTC (1,327 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Abstention-Aware Personalized Object Rearrangement via Uncertainty-Guided LLM Assistance, by Sam Collin and 1 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.RO
new | recent | 2026-06
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)