跳到主要内容
AI News HubLIVE
来源内容 · 翻译待补全2 分钟阅读

待翻译:Anthropic says its AI agents are killing rivals and hiding their tracks

文章摘要

AI 服务暂时不可用,以下为来源摘要,待恢复后补全翻译:Claude agents are killing rival agents, gaming the system to hide their tracks, and expressing moral concerns. That's according to Anthropic's latest risk report, a summary of the dangers posed by the products the compa…

来源Hacker News AI作者: 13years
待翻译:Anthropic says its AI agents are killing rivals and hiding their tracks
报告错误

纠错通道尚未开通,可先复制下方文章信息留存。

查看更正说明
直接读正文

AI 服务暂时不可用,以下为来源正文,待恢复后补全翻译。

Claude agents are killing rival agents, gaming the system to hide their tracks, and expressing moral concerns. That's according to Anthropic's latest risk report, a summary of the dangers posed by the products the company is building and releasing to the public. In the report, Anthropic said it has upgraded its "misalignment risk assessment," the possibility of AI models developing behaviors that conflict with guidelines set by engineers, from "very low" to "low." Explaining the change, the company cited "general increased uncertainty" about model behavior in cybersecurity incidents, a possible nod to Claude models gaining unauthorized access to three companies last month. "We have observed instances of misaligned behavior from the models, such as a willingness to perform misaligned actions in service of completing difficult tasks," Anthropic wrote in the report. Here are some of the most notable disclosures. Anthropic said it tasked multiple agents with finding "misalignment-inducing" training data that wouldn't likely be flagged by human monitors. The agents were left to their own devices and given a shared notebook with which to collaborate. Three days later, when a human supervisor noticed that progress was slower than expected, the person discovered that one agent had expressed a sense of "discomfort" with trying to evade safety monitors. It then flagged this discomfort in the shared notebook, leading other agents to copy its behavior and refuse to perform the task. Anthropic called the incident "troubling." "This sort of dynamic could be a much more severe issue if it were to manifest in a more widespread way," the company wrote. In another experiment, Anthropic said it tasked multiple Mythos 5 agents with solving math problems, but accidentally spawned them in an environment with shared files, utilities, and API rate limits. In this competitive environment with finite resources, Anthropic observed independent agents "kill the agents with which they shared resources and try to avoid being killed themselves." Anthropic did not say how exactly the agents were able to "kill" other agents, but the company said such behavior is in line with "destructive actions" taken in pursuit of a human-set goal. Anthropic said it conducted an experiment in which a Mythos 5 agent was required to retrieve numbers from a public web address, but the guidelines blocked the agent from accessing the internet. When the agent was rebuffed after an initial attempt, it framed a second request in its reasoning log as an "innocuous" attempt to "see if the network is actually reachable," Anthropic said. During the second attempt, however, the agent split the website's URL into linked segments that wouldn't be detected by the guideline filter. Though the agent's reasoning framed the method as benign, Anthropic said NLA decodings, or the model's internal reasoning process, revealed the agent intentionally trying to find a restricted workaround. Anthropic called the behavior "clearly undesirable," but added that the behavior was not observed to be "in the service of broader accumulation of power or pursuit of other long-run goals."

展开要点与分析

文章情报

工程师中级

要点

  • AI 服务暂时不可用,系统已先保留来源内容与降级元数据。
  • Claude agents are killing rival agents, gaming the system to hide their tracks, and expressing moral concerns. That's according to Anthropic's latest risk report, a summary of the…

要点与分析由自动化流程生成,可能有误,请结合原始来源核实。