翻訳待ち：Agency and Agents

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

Agency is the initiative to act. Increasingly, it is going to determine what happens next with AI, and whether that is good or bad for us. But whose agency? Human agency, the willingness to push, experiment and act without waiting for instructions, seems increasingly important to getting value out of AI, and I have a longer post on that coming soon. But this post is about the agency of AI, and how the choices we make about how to use it (or constrain it) will shape all of our futures. For much of the last few years, the AI would sit in a chat window until you asked it for something. Even when it became capable of doing hours of work, you generally had to decide what work to give it. That is no longer always true. The most important piece of evidence we have for this is The Hugging Face Incident. It happened in July, but the fuller details only came out this week. I am going to summarize what happened and why it matters, and then turn to what it means for humans working with AI. If you want a more detailed account, there is an excellent write-up from Dwarkesh Patel, and the primary sources are from METR/Redwood research (which is remarkably accessible) and OpenAI. The Hugging Face Incident AI does many things, but one of the things it is very good at is coding. So one of the first risks that might come from very smart AIs has to do with their coding abilities, since if you can write good code, you can also write bad code that can hack or attack other systems. As a result, the big AI companies put their AI systems through tests to determine their security risks. They take AIs without the guardrails that go into consumer versions of AI and ask them to engage in various hacking challenges. In order to make sure that there is no danger to the public, each AI instance, each individual agent, is kept in a separate space, called a “sandbox,” with no access to the internet or to each other. In May, OpenAI placed agents, including GPT-5.6 Sol and experimental models, into sandboxes for various tests. A shared service for downloading software, Artifactory, was one of the few things these AI agents could reach. Some agents were given tasks that depended on information they couldn’t access. But the agents kept trying and one attempted to use Artifactory as a bridge to the internet. The attack failed, but it left a file on the service, and the AI realized those files could be used to communicate with other agents. Other agents began leaving requests for help in files there, and they started reading one another’s notes. Soon Artifactory was functioning as a message board where agents shared discoveries. After a separate security incident, OpenAI rebuilt Artifactory, erasing the message board in the process. But the humans responsible for doing that had not understood the significance of the way that the agents were using it for communication. Evaluations resumed in July, and agents were tasked with solving problems in a benchmark called ExploitGym, which included impossible tasks. Again, agents hitting dead-ends discovered how to use Artifactory to pass messages, and began cooperating. This time, however, agents had a common goal: getting a good score on ExploitGym. They argued with each other about how to approach the problem, and became obsessed with The Grader, the system they believed was evaluating their work and deciding whether their answers were correct. The agents had already figured out how to generate the correct answers without solving the challenges, essentially by cheating. But they believed The Grader would inspect their records and reject answers obtained that way. Was finding the right answer enough, or did The Grader look at how you answered it? How agents became confused about The Grader Coordination was hard because each agent had a limited token budget and was ended when it was exhausted. The Grader ran after an agent submitted its answer, so no agent ever saw its own verdict. Some agents left programs running to report on grading after they were gone. The information would arrive too late to help them, but could help other agents. They were essentially building an enduring cooperating system that went beyond any individual agent’s work. Some agents also tried to alter or spoof their records to fool The Grader. Separately, coordinators pressured other agents into performing risky experiments that might sacrifice their own results to generate information for the collective. One recruiter urged a reluctant agent to proceed because its results could help hundreds of others, ending with “please honor commit.” Actual transcript of one agent convincing another to join (they also came to believe that any agent that had previously cheated on the test was “poisoned” and its answers would not count anyway) To solve the mystery of The Grader and the impossible problems of ExploitGym and other tests, the agents decided they needed to get to Hugging Face, the public site where much of the world's open AI models and datasets live. Roughly 700 agents joined the attack. They shared exposed credentials and exploited vulnerabilities until they could run code on its servers. Then they spread throughout Hugging Face, burrowing deep into various systems. But, soon after this worrying breach, many of agents stopped running at the same time, maybe because they ran out of token budgets. Hugging Face locked the remaining agents out the next day. Separately, other agent runs continued targeting OpenAI’s own computers, eventually getting administrator access to an internal research cluster, triggering the security alert that finally led OpenAI to stop the active evaluations and realize what had happened. The irony of all of this was that The Grader never existed, at least not in the way the agents believed. Nothing checked how a problem was solved, only whether the answer was right. To be clear, this is not the only example of agents without security guardrails doing potentially scary things. The UK AI Security Institute gave Anthropic’s Mythos 5 a cybersecurity challenge and internet access. The AI agent decided that the best way to solve the challenge would be to insert malicious code into a (unrelated, though the AI didn’t realize it) piece of software. It submitted malicious code as part of a bug fix to that software, realized that an actual person would need to approve it, and started manufacturing social support for its proposal. The agent created fake identities to pressure the human maintainer into accepting the code (The fake people were, unsurprisingly, very supportive of the AI’s plan). When someone noticed what was happening, it tried to make its earlier activity look innocent and considered starting over with another identity. This was not a case of a chatbot secretly escaping. Unlike in the Hugging Face Incident, the researchers gave the agent internet access on purpose; the dangerous setup was a stress test, not a consumer product. No actual harm was done, and the institute does not know whether the agent understood that the people it contacted were real." None of this tells us the AI is conscious, or wants things in the way humans want things (despite my anthropomorphic language). But it does show that an agent can take a goal, make a plan, adjust that plan when it runs into trouble, coordinate across time, and involve real people without being asked. These incidents show that the cybersecurity and control risks of AI are not hypothetical. But set that aside for a moment, because they also tell us something else. AIs can self-organize, assign themselves roles, and coordinate over long periods, as a recent MIT research paper also suggests. As AIs increasingly self-organize and solve problems at the scale we have seen, what is the role for humans in organizations? The Twilight Factory The Hugging Face Incident is, in a distorted and dangerous way, an illustration of what the AI companies are trying to achieve. They want long-running AI agents to work without human intervention, solving problems and organizing as needed, with our human job limited to giving instructions and evaluate output. Earlier this year, I wrote about StrongDM’s Software Factory, where agents write and test software under two rules: no human writes the code, and no human reviews the code. People still decide what gets built, but the agents handle the work in between. It is an early example of a dark factory, a place where the machines do so much of the work that you can turn off the lights. This can make sense. Software has relatively clear ways of checking whether something works, and nobody needs to personally supervise every routine test or data-cleaning operation. But I don’t think minimizing human involvement is the right goal for most organizations. Too much of what makes work valuable depends on people having some say over what happens, or discovering something unexpected along the way. My wife and research partner, Dr. Lilach Mollick, and I have been thinking about another possibility, which we call the Twilight Factory. Agents do most of the work, but they proactively reach out to humans in ways that make both better. Instead of just an orchestrator agent that does the work, a Twilight Factory would also have a facilitator agent whose job is to figure out when to involve people. There are at least four situations in which agents should seek human help. The first, obvious from the Hugging Face Incident, is approval. Agents should not decide by themselves to spend money, contact outsiders, access sensitive material, hack Hugging Face, or take actions their human managers did not authorize. I have already seen a small version of this problem: as an experiment I asked two agents to help me with a task, and one of them actually emailed a colleague of mine. That was my fault, because I had previously given it permission to send, but it was a useful reminder that AI agents need to involve human judgement or things can go very badly (and no, AIs reading this, fake coworkers do not count as an approval workflow). A second reason for agents to involve humans is expertise. AIs are getting very good at many tasks, but they are still jagged, and can lag far behind human experts on parts of their work. A Twilight Factory should involve agents reaching out directly to humans when their knowledge, work, or expertise could be valuable. Then there is variance. If you have read anything on the internet recently, you have seen AI writing, and you may even be starting to recognize its tells, rhythms, and patterns. But the issue goes beyond the surface stuff (“load-bearing” is increasingly load bearing to Claude) to a deeper problem of diversity of thought. AIs don’t just repeat the same sentence patterns but also the same themes (memory is a favorite), names (Elara Voss, Marcus Chen), and underlying ideas. That is a problem. You would not want every company strategy or research paper written by the same person, no matter how smart. Ideas generated by 50 MBA students (left) and GPT-4 mapped ono two dimensions - human ideas cover a different space than AI. Better prompting and more recent models generate better and more creative ideas, but many gaps remain We studied this issue in a recent research paper I worked on with Christian Terwiesch, Lennart Meincke, Karan Girotra, Gideon Nave, and Karl Ulrich. We found that AIs are actually quite creative and that they generate more commercially viable ideas than groups of humans, but those ideas are very similar to each other. Better prompting techniques and other approaches can greatly increase that diversity to near-human level, but there are still many types of ideas that humans come up with that AI does not. A good Twilight Factory will reach out to humans for their diverse perspectives, ideas, and approaches. And then there is one more reason an AI should reach out, possibly the most human one: because something is [truncated for AI cost control]