待翻译：Software Factories Are Distributed Systems

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

Every abstraction here is taught through Gas City, a software factory SDK I run and help maintain. An agent in my fleet fixed a scoring-integrity bug (yay!). It made the change in its own branch, added a regression test, ran the suite, passed automated review, recorded a verdict of pass, and closed the work item. Every step reported success, and every step really did succeed. Nothing merged the branch (not yay). The commits sat there, three ahead of main, with the new test present on the branch and absent from the codebase everything else was reading. This was unfortunately an embarrassingly common occurrence for me. Elsewhere in the same rig (basically a repo plus a work-items database in Beads), audits kept reading main, finding bugs whose fixes had already been written, tested, reviewed, and closed on branches nobody merged, and filing them again. The working code existed, but the system never folded it in. Some of this amounts to a game of Agent Telephone. An agent will do what you tell it to, sometimes, except when it doesn’t, or when it helpfully decides you meant something adjacent. But that wasn’t the problem here. The agent did the work it was asked to do and reported accurately on what it had done, and the missing step was outside the agent entirely. Nothing owned the transition between producing a valid change and making that change part of the codebase. That meant a closed work item and merged code were allowed to stand in for the same fact when they are very much not the same fact. One layer observed something, another layer treated it as authority, and nearly every failure below is a version of that. Agents fail on their own too, and later sections have plenty of them: workers skipping a check the protocol told them to run, closing work the instructions said to leave open. But those aren’t failures you fix by writing a better instruction, which is the actual problem. A prompt can’t hold an invariant. So the factory around the agent has to keep track of who is allowed to act, whether an effect has already happened, what artifact was actually verified, when work is really complete, and what to do when part of that sequence dies halfway through. None of those guarantees come from the model or its toolkits. They also do not belong uniquely to a workflow engine, queue, or agent runtime. They emerge from how all of those pieces interact. An old problem with a new worker Not every agent deployment needs a factory around it. A local assistant that reads a repository, proposes a patch in an interactive session, and exits has one process, one human, and very little durable coordination state. If it dies, the human restarts it and mostly loses some minor convenience. The failure model changes once 1) the work has to outlive the process doing it, 2) multiple workers can act concurrently on shared or versioned state, 3) components can fail independently, 4) external systems commit effects asynchronously, or 5) verification and publication happen in separate places. At that point you have acquired the usual distributed-systems problems whether or not you’ve thought to wrap your head around it that way: stale authority, duplicate effects, lost updates, split-brain records, and partial failure, to name a few. Thinking about software factories this way isn’t anything new. Osterweil argued in 1987 that software processes are software too. Choi and Scacchi described “the software infrastructure for a distributed system factory” in 1991, treating the coordination plane as something that had to be engineered in its own right. The CNCF’s Secure Software Factory reference architecture supplies much of the contemporary vocabulary. What autonomous agents change are the characteristics of the worker. The mechanisms in this essay are not new. Fencing, leases, idempotency, reconciliation, and conditional writes have decades of distributed-systems history behind them. What surprised me was how little scale it took before I needed them. You can get stale authority, duplicate executors, conflicting effects, and split-brain records with three coding agents sharing one repository. “Distributed system” starts sounding grandiose right up until one worker dies, its child keeps editing, and the retry starts another one. The interesting boundary is not fleet size. It is the moment workers can act independently on durable or shared state while their supervisors, observations, and external effects can fail separately. Those older systems mostly coordinated deterministic tools and human developers who could, at least in principle, be asked what they had done and why (instead of “idk Claude said to do this,” unless Claude happened to be that one guy hoarding the company COBOL knowledge). A compiler does not confidently explain that it compiled the program when no binary exists. An agent can absolutely tell you it completed a task whose authoritative effect never happened, often in beautifully aggravating detail, and then apologize and do it again. That makes the distinction between what a worker says happened and what the system can prove happened much more important. It is one reason current software factories have arrived at similar decompositions from different directions. OpenAI’s Symphony orchestration, Cloudflare’s issue-triage factory, and Vercel’s factory for the AI SDK repository all separate some version of durable work state, scheduling, disposable workers, and gated publication. Vercel, for example, records runs as success, flawed, blocked, or manual, and only success ships. Vercel is a useful comparison for another reason: the thing its factory produces is not the factory itself. Its agents work on the AI SDK, including bug fixes, features, documentation, and backports, while a human retains the merge boundary. Four weeks after launch, Vercel reported that factory agents were authoring 25 to 35 percent of the pull requests merged each week. My own factory has the same separation between orchestration infrastructure and the work being produced. I run the same substrate across evaluation infrastructure like CodeScaleBench, research tooling, developer tools, this website, and Gas City itself. Gas City shows up disproportionately in the incidents below because it is the one place where I control both the orchestration layer and the code being changed. That makes it particularly useful for fault injection, recovery experiments, and following a failure all the way from worker behavior to authoritative state. The incidents are therefore sampled from the part of the system where my observability and experimental control are strongest, not because the factory exists primarily to maintain Gas City. And this is where the apparently straightforward architecture starts getting less straightforward in practice. Operate enough agents for long enough and you discover just how many layers in the factory can quietly collapse “the attempt stopped” into “the work is done.” Most of this essay is about those places. The machinery that fixes them is old: fencing, leases, idempotency keys, prepare-then-commit, reconciliation. I did not have to invent any of it. Two things are different now. The worker reports on itself in fluent prose, and it is the most articulate component in the system. Every question about evidence gets harder when the part best able to explain what happened is also the part with the least authority to say so. And the fleet is continuously building a second codebase that hasn’t merged yet: dozens of live worktrees and unmerged branches, invisible to anything that reads merged state. Two agents heading for the same interface don’t collide until one of them lands. Everything below is how I found out about all of it, by watching my factory break. My factory Reliability abstractions can be particularly eye-glazing, so I’ll introduce each one through the thing that actually broke in my factory and generalize from there. That means a short vocabulary lesson first. Gas City is an open-source SDK for running fleets of coding agents (gastownhall/gascity). It gives you the building blocks rather than one fixed design: a durable work record, persistent agent identities, bounded worker sessions, dispatch, scheduled jobs, and messaging between agents. I help maintain it, and I run an installation of it that builds and repairs Gas City itself, which is where the incidents below come from. A bead is the unit of durable work, from Beads, the dependency-aware work record underneath. Nearly everything in the city is a bead: tasks, mail between agents, workflow steps, and the live agent sessions themselves. A bead has an id, a status, an assignee, labels, and metadata (my provenance PR was merged yay!), and it lives in a store that outlives every process that touches it. A rig is a project workspace with its own bead database and its own id prefix, so a bead’s id tells you which rig owns it. A worker is an agent, usually one of a pool of interchangeable slots, that claims a bead and does the work in its own worktree. A claim is how it takes ownership, and the mechanics get a whole section below, because that is where authority is won or lost. A formula is a reusable multi-step workflow, and an instance of one is a molecule whose steps are themselves beads, so a workflow’s progress is visible in the same store as the work. An order is a scheduled job that fires on a cadence, basically a cron job, and my city runs about a hundred of them: compactors, sweeps, and reapers, the repair loops that find and fix divergence. The mayor is the top-level coordinating agent and the handoff point to me. Four responsibilities a factory must not confuse A factory has four responsibilities, and Gas City keeps them separate, which is why it’s easy to see when one gets confused for another. The bead store is the work ledger, holding durable facts: what work exists, who owns it, what came out of it. A formula is the procedure, deciding what happens next: ordering, waits, retries, cancellation. An agent is a worker, making changes in worktrees, repositories, and external services. Pool sizing and admission are the control plane, deciding what runs at all. None of the four can stand in for another. The control plane sets policy: what may run, at what priority, with what share of the fleet. The work ledger holds durable facts; recovery starts there, not from worker memory. The procedure layer holds ordering, waits, retries, and acknowledgements. Workers produce effects and are the only layer that touches external systems. The fence sits at the external boundary, where a mutation becomes authoritative. Most incidents are one of these layers being read as evidence for another: a running process taken for a valid claim, a completed procedure taken for a published change. Mixing them up is where things break, usually because one layer’s evidence gets treated as another layer’s authority. A running process doesn’t prove the work is still assigned to it. A completed procedure doesn’t prove its branch ever landed. A closed work item doesn’t prove the result went anywhere at all. It’s completely fine for agents to be nondeterministic, but for reliable work throughput the authority over what they produce needs determinism. Recovery has to start from durable facts rather than from what a worker remembers, what a process happens to be doing, or what a procedure thinks it asked for a while back. The failures below are different versions of that rule getting broken. More importantly, each version can be turned into a rule, deliberately broken in a test, and checked. The work outlives the worker One piece of work has more identities inside it than a design usually bothers specifying, and letting any two of them blur together is an incident waiting to happen. Seven are worth naming: the work itself, the state of the code the attempt started from, the ownership epoch that [truncated for AI cost control]