Steering AI Beyond Automation

By Gabriel Unger and Arvind Narayanan. Unger is an economist at Stanford’s Digital Economy Lab. One of the most fundamental questions for the field of AI today is: to what extent is the project of AI purely about automating human labor? The dominant Silicon Valley program for AI right now, with both industry and academic faces, answers: “almost entirely.”1 There is an implicit technological determinism: that AI has a deep internal logic that favors automation. To keep pushing in this direction is to move towards economic growth, prosperity, and profit. The vision of AGI is the successful completion of this project: to create AI systems capable enough to autonomously do everything a human can do. There is an alternative view: that AI is a technology (or class of technologies) that, like any technology, can develop in different directions. Total automation is one direction. But there are others, including directions that focus more on human augmentation, and increasing the collaborative and collective power of teams of humans and AI models working together. This view — our view — is not that automation is never good. Clearly sometimes it is, and clearly sometimes some subset of people try to oppose it out of the narrowest short-term self interest, at the expense of broader social and economic progress. But we think AI can also augment human workers in many interesting ways, that some of the maximalist attempts to completely automate all human labor are deeply unpromising on both technical and economic grounds, and that right now too much of AI research is devoted to the latter at the expense of the former. The bias toward the automation agenda is partly because of economic incentives, but partly because of some deeper intellectual and cultural aspects of computer science as a field. The alternative view has deep intellectual roots in the history of computer science, especially the subfield of human-computer interaction. Moreover, there are compelling technical and economic reasons to pursue it now. A small but growing number of businesses are now coming around to it, after witnessing the economic disappointments of attempting to implement the first program. Despite this, it remains a minority position today, poorly understood, and disorganized. Above all, computer science as a field today lacks a clear articulation of a systematic program for the architecture of AI that would further this agenda. In this essay, we briefly discuss what is at stake here, the history of this debate in CS, the status of this debate today in economics and elsewhere, and the automation agenda in the culture and curricula of CS in the university. Above all, we sketch in Section 1 a few research directions for AI that push in the direction of human-AI collaboration. Designing AI for true collaboration requires solving a very different set of technical problems than today’s models and agents, problems that are often ignored, or not even understood to be problems. The specific directions we sketch are intended to be tentative and experimental. But the point is to explicitly conceive of a project for AI that differs from the dominant Silicon Valley one, and moves beyond criticism to the positive articulation of an alternative direction. Our hope is that this can serve as a seed for a much broader, sustained, experimental, and collective research program. In Section 2, we discuss the high level motivation for our project, and in Section 3, we give a brief history of some of these ideas. In Section 4, we put this all into the context of the debates today about AI and automation in economics and in policy, and in Section 5, we outline a bottom-up, market-driven approach to steering the direction of AI development and deployment. 1. A Research Agenda for Collaborative Agents A serious research program for human-AI collaboration elevates novel technical problems that the automation agenda for AI is more likely to ignore or treat as relatively unimportant. We present four such illustrative directions. Problem #1: Can we produce white-box agents for various domains of human labor? The dominant paradigm for AI models and agents is “black box”: the system is evaluated based on the final output it produces. The intermediate steps are generally not revealed to the user, may be mysterious even if revealed because they depart greatly from how people approach the task, and are not treated and optimized as an aspect of task performance. One reason for the black box paradigm is the absence of process traces in model training. If we train a new AI model on a thousand paintings from the Renaissance, but we have no data on any of the many intermediate steps and revisions involved in the creation of any given painting, we can’t expect anything other than a black box model. Maybe it can generate the image of a new painting in a similar style, but it will not be able to actively talk us through the steps involved in creating such a painting. What we instead call “white box agents” are AI systems that are epistemically more legible to their human users, and explicitly reveal their intermediate processes (sources, assumptions, methods, uncertainties, process artifacts). White box agents could open the door to all kinds of more direct forms of human-AI collaborations: if humans can see the intermediate processes, they can directly interact with them. The successful development of white box agents will require tackling many technical sub-problems. Chiefly among them: capturing process traces for a much broader range of human labor. The past few years of AI model training and deployment have had a disproportionate focus on software coding. There are many reasons for this, but a big one is that we benefit from the very public availability of a massive number of coding repositories, which include traces of every step of the process of software development. But for so many of the other important professional occupations in the economy today — law, medicine, many sciences — no such magical repositories of publicly available, free, comprehensive and easily codifiable process trace data exist. Here’s a thought experiment. Imagine a counterfactual world in which open-source software didn’t exist, and coding agents (out of necessity) were trained to directly output compiled software binaries instead of source code, in a black box way. We think this capability would be experienced as dystopian instead of liberating by software engineers and it would have the effect of replacing them instead of amplifying their abilities. The shift we advocate is the equivalent of introducing code repositories and coding agents into this world — and doing so in every domain that currently lacks them. There is a quiet gold rush to capture process traces for a broader range of occupations to feed into AI models, by both companies entirely dedicated to the purpose (Mercor) and legacy tech companies ruthlessly trying to study their own employees, the better to replace them (Meta). Paradoxically, this is all often done in the service of the automation paradigm: come up with training data to make more black box agents in more domains. But it could also be vital in better enabling the development of a new generation of white box agents. Problem #2: Can we develop agents fast enough for real-time interaction? A basic dynamic of active human collaboration is low latency. Imagine two people having a real conversation. If one person has to leave the room to think for 20 minutes every time you ask him a question, you’re no longer having a conversation; at best maybe you’re delegating to him. Models that can work at a speed that more naturally resembles human interaction might better enable forms of deeper human-AI collaboration. Today, we are forced to pick between chatbots that can answer questions quickly but superficially, and agents that can perform tasks thoroughly but break the flow of conversation. We don’t have to be stuck with this tradeoff. This sounds like a simple goal, but it implies a comprehensive set of technical challenges, that potentially span everything from model inference to retrieval methods, shells, browsers, tools, and more. How can we build AI systems that achieve latency low enough for collaboration? What parts of the agent stack could be changed, or need to be changed, to achieve this? Low latency is particularly promising because it could enable workers to stay in a state of flow. Flow is a concept from psychology that describes people’s experience when performing activities such as rock climbing or chess that people are drawn to for intrinsic reasons. It is a state of altered experience in which the performer forgets time and fatigue, and is immersed in the moment. Most programmers are familiar with flow, and it is a big reason they are drawn to programming. Working in a state of flow requires clear and immediate feedback — that is, low latency. So far, agents have weakened our ability to achieve flow. Many programmers report that the shift from manual programming to supervising coding agents has resulted in a loss of flow, cognitive overload and burnout. But our hypothesis is that it is possible to redesign agents to not only restore flow, but even increase people’s ability to achieve flow. Problem #3: Can humans and agents continually work together, without the constraints of the turn-based paradigm? The basic model of human-AI interaction, under the automation paradigm, is delegation: I tell my agent what to do, walk away, then come back later to verify. Many versions of “human in the loop” still come back to this; the human is technically in the loop, but just standing at the starting line and then waiting at the finish line. This is quite different from a model of collaboration, where the human is actively participating in the process, and continually steering the agent as it works. A primitive version of continual steering exists in mainstream coding agents today, but is limited to giving additional instructions to the agent as it works. It is a slight tweak to the paradigm of the user and the agent alternately taking turns, rather than a departure from it. If the user tried to edit the same file as the agent, it would get confused and all bets are off. True collaboration poses a novel set of technical challenges. Systems will need to display visible states that change continuously instead of turn-by-turn, have shared workspaces, and accommodate interruption, parallel work, simultaneous editing, and other collaboration patterns. This is especially important in domains where it is difficult to precisely specify the task ex ante, or verification is complicated and perhaps bound up in implementation. Meeting this challenge would require designing architectures of interaction that allow continuous control without melting down under the interruption. Problem #4: How can we evaluate, and optimize for, human-AI collaboration? Most AI benchmarks today test how good a model is at completing a task on its own, not on how well a model collaborates with a human. (This is likely part of the reason for the gap between benchmark performance and real-world usefulness). Evaluating the joint efforts of AI-human teams is hard. This is a relatively simple challenge to articulate, but non-trivial to make progress on. It is more difficult to understand how to best evaluate AI-human collaboration, and it raises a complex set of questions that at first blush might seem uncomfortably subjective or confusing. How much disagreement between the AI model and the human is appropriate? How deferential should the model be, and how much control should the human have? How much space does a productive collaboration have for each party to learn from the other? How do we think about short term vs long term success of a team? What are the most objective ways to evaluate collaborative success, and what are [truncated for AI cost control]