Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models
This paper introduces DROPJ, a human-centered method for safe training and deployment of agents in safety-critical environments with unknown dynamics and no suitable reward function. DROPJ first learns a world model from prior real-world trajectories, then has a human play in the simulator to generate informative simulated trajectories. Preferences and justifications are elicited from humans on trajectory segments, which are used to train a reward model. The agent is then deployed using model predictive control with the world model and reward model. Experiments show that generating informative simulated trajectories significantly reduces computational cost and improves deployment performance, with preference feedback outperforming other types, and safety justifications enhancing safety.
-->
[Submitted on 14 Jul 2026]
Title:Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models
View a PDF of the paper titled Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models, by Ilias Kazantzidis and 3 other authors
View PDF HTML (experimental)
Abstract:We address the problem of safely training an agent policy and deploying a good and safe policy, in settings where the environment dynamics are unknown and no suitable reward function is available. In the context of safety-critical environments, we consider traditional reinforcement learning impractical and resort to the resource of human input. We introduce DROPJ, a human-centred method for both safe training and deployment. We first learn a world model (a learned simulator) from a dataset of prior real-world trajectories. A human then plays the game in this learned simulator to extract several informative simulated trajectories. From these, we sample pairs of simulated trajectory segments and elicit from a human their preference over these segments, as well as a reason (justification) for their choice. We then train a reward model from these justified preferences and use it, together with the world model, to directly deploy the agent using model predictive control. Running real-user experiments, we find that generating informative simulated trajectories from a user significantly reduces the computational cost during training compared to other strategies, and can also improve the performance during deployment. In the context of training within a learned simulator, we show that the use of preferences rather than other types of feedback substantially improves the performance during deployment. We further demonstrate that safety justifications accompanying preferences can significantly enhance safety or prioritise user-prescribed aspects of safety associated with them during deployment.
Comments: 42 pages, 18 figures. Extended version of a paper presented at ICAART 2026; submitted for consideration in the ICAART 2026 post-publication selected-papers volume in Lecture Notes in Artificial Intelligence
Subjects:
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2607.13172 [cs.AI]
(or arXiv:2607.13172v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2607.13172
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Ilias Kazantzidis [view email] [v1] Tue, 14 Jul 2026 18:22:14 UTC (4,713 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models, by Ilias Kazantzidis and 3 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new | recent | 2026-07
Change to browse by:
cs cs.LG
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)