Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety
A new study shows that if attackers strategically choose when to attack, the measured safety of AI control evaluations drops significantly. The researchers decompose attack decisions into start and stop policies, and demonstrate on BashArena and LinuxArena that both policies reduce safety by 20 to 28 percentage points at a 1% audit budget, without changing the underlying attack capability. Existing evaluations may overestimate safety, and the study recommends incorporating attack selection for more realistic estimates.
[2606.06529] Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety
[Submitted on 3 Jun 2026]
Title:Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety
View a PDF of the paper titled Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety, by Catherine Ge-Wang and 5 other authors
View PDF HTML (experimental)
Abstract:An attacker that strategically chooses when to attack is much harder to catch than one that attacks indiscriminately. AI control is a safety framework for deploying capable but untrusted AI agents under the oversight of a weaker, trusted monitor and a limited human audit budget. Control evaluations stress-test these protocols by pitting a red-team attack policy against the blue-team monitor, but current evaluations typically assume attackers that do not strategically select when to attack. We study this capability, attack selection, in agentic settings by decomposing attack decisions into a start policy, which decides when an attacker should attack, and a stop policy, which decides when an attacker should abort an ongoing attack. Across two agentic settings, BashArena and LinuxArena, both policies substantially lower measured empirical safety without changing the underlying attack capability. At a 1% audit budget, our start policy reduces safety by 20pp on both BashArena and LinuxArena, and our stop policy reduces safety by 20pp on BashArena and 28pp on LinuxArena. These reductions should be interpreted as upper bounds on the effect of attack selection. Existing control evaluations may therefore yield overly optimistic safety estimates against selective attackers. We recommend that future evaluations, system cards, and safety cases elicit attack selection to produce more realistic safety estimates.
Subjects:
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2606.06529 [cs.AI]
(or arXiv:2606.06529v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.06529
arXiv-issued DOI via DataCite
Submission history
From: Tyler Crosse [view email] [v1] Wed, 3 Jun 2026 11:04:59 UTC (5,255 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety, by Catherine Ge-Wang and 5 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new | recent | 2026-06
Change to browse by:
cs cs.LG
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)