待翻譯：Wild AI-related reliability incidents are coming

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

Recently, two AI-related pieces of content caught my attention. The first was the blog post On-Call is Now Theatre by Boris Tane. He argues that AI agents are now capable of doing the majority of on-call work that is currently being done by humans, and that we should have AI agents act as first responders. Only when an AI agent isn’t capable of remediating the problem should a human actually be brought in, and the agent should be the one to page the human. As he puts it: We need software that watches itself, triages its own alerts, investigates its own incidents, fixes what it can, and escalates to a human only when it hits something genuinely novel, with the evidence already assembled. Put your AI agents in the worst on-call rotation imaginable, then give them a tool to page a human. Developers stop being the first responder, and step in only when an agent genuinely cannot figure something out. Tane doesn’t think that companies will really start putting AI agents on-call (“Most teams won’t do this”), but he believes it should happen, and he’s started a company based on this premise. I like to think of putting AI agents on-call as equivalent to using AI agents to implement control system automation. Because, after all, that’s what operations work is: it’s taking control actions to keep the system in a healthy state. Now, AI agents are extremely complex software systems. I’d argue that they are the most complex software systems that we humans have ever built. That complexity is both good and bad. Ashby’s Law teaches us that the larger the set of system states that you want your control system to be able to handle, the more complex that it needs to be. It’s this complexity that makes it possible, in principle, to apply AI agents to solve a generic control problem like this. On the other hand, the more complex a system becomes, the more difficult it is for a human to reason about the system’s behavior. That’s fine when the system is healthy, but if your now-even-more-complex system gets into a state that the automation can’t handle, that can make the problem even worse. Indeed, it’s precisely the unexpected behavior of complex control systems that contributes to the worst complex systems failures (see also: Air France 447, Boeing 737 MAX accidents). And that brings me to the other piece of content I saw recently: the OpenAI talk at BlackHat (h/t David Blank-Edelman). Yes, it’s a 37 minute talk, but I encourage you to watch it.