A Dynamical Model of AI Governability

Introduction#

Deceptive alignment stories depend on races: misaligned AI systems get ahead by acting deceptively while supervision lags too far behind to uncover their duplicity. We depend on AI assistance to keep pace in the oversight race, and they help us keep pace as long as we're able to catch and fix any tendencies they have to act uncooperatively. If we fall behind in the oversight race, we lose the tools we were relying on to keep pace, leading to a stable loss of control.

The outcome thus depends on competing quantities - advanced AI will be better at evading misbehaviour monitors, better at building misbehaviour monitors, and whether it actually helps to build misbehaviour monitors is complex and path dependent. It is very hard to resolve questions of where this is going via verbal analysis - we really need a quantitative model to understand the system. In this post we'll be building just that - a quantitative dynamical model of AI governability. Our model is in the same family of the AI Futures model; it's an ambitious attempt to model an important system (in their case, AI capabilities take off; in our case, retaining control of advanced AI), and while we have spent considerable time critiquing and refining the model and finding empirical anchors for quantities where possible, by necessity it is less rigorous than would typically be found in academic literature.

Given the challenges involved, what are the benefits of creating a model like this? Firstly, it gives you predictions of the outcomes of building advanced AI - do we succeed in building systems that support our aims or not? There is enough uncertainty in multiple parameters that the model shouldn't be viewed as picking out a particular, likely trajectory. Rather it can help identify key uncertainties that flip the prediction, and points of intervention that may also exert significant influence.

More ambitiously, we think there would be considerable benefits to a robust AI takeover early warning system. Multiple factors make AI governance a very difficult problem:

AI takeover is a plausible if uncertain outcome of continued AI development, and if we were truly headed toward takeover extreme actions would be justified to avoid it

The benefits AI brings promise to be very large, so actions to control AI can have huge costs and attract fierce opposition

Building the option to, for example, completely halt AI development requires centralising a huge amount of power in the hands of the regulator

There's widespread disagreement about which risks are most pressing, and how pressing those risks are

If it were possible to build a robust early warning system for loss of control, it would go a long way to mitigating these difficulties. If we know with precision whether we are tracking along a safe or unsafe trajectory, we can be more confident our interventions are sufficient to keep us on the safe path while at the same time avoiding doing too much and losing a great deal of value from AI as a result. Even if AI development poses grave risks, information about where we are tracking can still be shared freely, unlike control over that development, and well informed people everywhere will make choices based on that information. In this way, an early warning system supports powerful decentralized responses to risks from advanced AI, which can complement or even partially substitute for centralized control of the industry.

We do not claim we have built such a system! However, any early warning system necessarily depends on a predictive model of outcomes, given our current best guesses about the state of the world. Thus we see this model as a proof of concept for a critical component of an early warning system - we've built and refined a model, we're presenting it here along with insights we took from the exercise and challenges we've not yet overcome, and we want to test it against criticisms from AI safety experts.

There has been a lot of work in the neighbourhood of this question, though to our knowledge there is no prior quantitative treatment of the oversight race. Carlsmith's analysis of automating alignment research explores topics like feedback loops, failure modes and observability of automated alignment research. Christiano's "What failure looks like" describes two broad civilisational failures under advanced AI: first, our ability to measure whether we are "getting what we want" erodes as AI systems grow more complex, and the systems themselves eventually stop cooperating with efforts to improve oversight. Second, AI development spawns poorly-observed 'greedy' patterns that seek to expand their own influence and end up dominating the system's behaviour. Our model explicitly includes both of these dynamics. Kulveit et al.'s Gradual Disempowerment describes gradual, systemic erosion of control driven by incentives, and stresses that aligning AI systems to designers' intentions is not sufficient to avoid this path. Our model is lightly opinionated on what kinds of AI failures we are trying to suppress, and it may be feasible to repurpose it to track gradual disempowerment concerns, though that's not within the scope of our effort here.

Hendrycks' "Natural Selection Favors AIs over Humans" describes how selection dynamics may lead to misaligned (or uncooperative) AI systems. Our model is a selection dynamics model, capturing competition between cooperative and uncooperative AI pools. Hendrycks argues that uncooperative AIs will probably propagate themselves better than cooperative ones. We treat this as an open question: our central estimate gives neither pool an advantage, with wide sampling ranges in both directions. In the appendix we also explore how a finer grained picture of selection dynamics could inform a more sophisticated model.

Everything in this post can be explored interactively: the basin explorer implements the full model with every parameter on a slider, one-click presets for the calibrations we discuss, and an outcome map showing which parameter combinations lead where.

The results in brief#

We model the workforce that builds future AI as two competing pools — self-propagating cooperative work, and self-propagating uncooperative work (henceforth just cooperative and uncooperative) — connected by a handful of rates: how effectively each pool can propagate itself, how often cooperative development leaks uncooperative systems, how much of current misbehaviour our detect-and-fix machinery covers and how fast it corrects what it covers (these are explained in detail in the next section).

Plausible futures: sampled model trajectories. Each band is the 85% spread (with a median line) over a Monte-Carlo ensemble that draws the uncertain parameters in our model. 'Good' outcomes involve keeping the line low in perpetuity; there is never very much uncooperative AI labour. We somewhat arbitrarily fix 10% as a 'high risk' threshold and analyse whether trajectories stay below this or rise above it; the exact level of 10% is not especially meaningful, but we do think policy should aim to maintain the uncooperative share below some threshold. Observed misbehaviour often does separate between good and bad trajectories, but the separation is often small.

1. Both success and failure to suppress uncooperative takeover are plausible. Under our central parametrization uncooperative behaviour settles at ~25% of the AI labour pool — two and a half times our 10% high-risk line. The outcome worsens toward takeover if accidental production of uncooperative AI ("leakage") runs above our estimate, if uncooperative systems find highly effective growth strategies, or if evading detection proves much easier than detecting misbehaviour — with the first two mattering more than the third. Marginal safety (an uncooperative pool just under 10%) is within our ranges of uncertainty on several fronts: leakage at half our estimate, uncooperative systems propagating ~30% less effectively than cooperative ones, or detection-and-repair 3-6 times more effective than assumed — whether because the world is friendlier than we estimated or because we invest more. Monitoring and suppression are cooperative; successful control depends on adequately observing uncooperative behaviour, and efficiently addressing it when observed. It's worth flagging a policy takeaway here: according to the model, safety is currently significantly underinvested.

1. The model has known shortcomings that pull in both directions. We calibrate our model as if all AI misbehaviour were self-propagating. That is not true today but we expect it will become more true in the future, but the identification of all misbehaviour with self-propagating misbehaviour is a crude model of this expectation. If it remains a fraction of all misbehaviour, the uncooperative pool stays small. On the other hand, dispersion and volatility in the ability of misbehaving AI to propagate its traits can greatly increase the risk of uncooperative takeover, as we discuss in the appendix.

1. Failure is generally smooth, but not necessarily slow. Our model tracks total AI labour $A$ in units of the current amount of human labour producing AI systems, $A/H_0$, so that 1 unit is "parity" — the point in time where AI labour facilitates a 100% speedup over unaided human labour ($A/H_0=1$). Our median "failing" path crosses a 20% rate of uncooperative AI behaviour at about $A/H_0=10$ (around 1.4 years separates $A/H_0=1$ from $A/H_0=10$ under naive exponential extrapolation). The fastest failures cross the 20% rate at about $A/H_0=2$, or 5 months under naive exponentials. Notably, there is no discrete "handoff" event which decides whether control is maintained or lost. Governability is the product of continuous trends — whether our aided ability to track misbehaviour keeps up or degrades. A handoff happens, but it is a long-running event that has to be continuously managed, not a setpiece we're building up to tackle.

1. Observed misbehaviour is a useful but imperfect indicator of uncooperative takeover. Generally, observed misbehaviour rises together with actual misbehaviour. Uncooperative labour undermines supervision, so this signal is attenuated, but it is usually not fully suppressed. As a result, observed misbehaviour rates give some indication of whether the trajectory of AI development is safe or unsafe, but its discriminating power is imperfect. In particular, every good path has a bad path that looks almost the same, though there are many bad paths without good counterparts. Because we often do see warning signals for adverse trajectories, there's a lot to be gained by having the ability to act on them; it is worth building the capacity for industry and policy makers to be attuned to signals about the risk of the current AI trajectory, and to respond to them quickly and with sufficient force.

1. Our model makes a number of predictions, and we find support for one of them. Our model predicts that re-scoring a model under a newer audit harness surfaces more misbehaviour, and we find support for this in Anthropic's released model cards. Our model also predicts that we will likely see a bend in the uncooperative trend - that is, the trend in uncooperative AI behaviour will bend upward as we transition from a human dominated to an AI dominated development regime. For more, see predictions and postdictions.

1. Some practical observations could significantly reduce our uncertainty. The model points to measurements that would be especially informative. Retrospective audits of older models could help estimate, with lag, what fraction of misbehaviour contemporaneous audits were able to catch. This comes attached with a recommendation: we should be aiming to preserve both auditing methods and AI development contexts in order to replicate older audits on newer models, and apply new methods to old models across a representative sample of contexts in which they were used. Estimates of the “reproduction numbers” of uncooperative behaviours via all known channels — including productive misuse, data poisoning, autonomous replication and vertical transmission (see also subliminal learning) — will help ground one of the model’s most important parameters.

1. Our model quantitatively reproduces some of the qualitative predictions in AI-2027, and disagrees with others. AI 2027 predicts that "Agent-4" will become misaligned (which we call "uncooperative") in a discrete event, and that there will be an attenuated but observable signal attached. Our model agrees and explains why the signal is hard to suppress. We qualitatively support the scenario's conclusion that under their implied high leakage, large AI development slowdowns are required to restore cooperative behaviour. We differ in our treatment of leakage; in AI-2027, leakage first rises sharply and then, under sufficiently careful development, falls sharply, while we treat it as a constant throughout. We think it changes too little in our model and too much in theirs; we discuss some ways to handle it better in the appendix.

Explaining the model#

What is being modelled#

We're not modelling whether the outcomes are ultimately good, all things considered. We're modelling whether AI that produces the next generation of AI becomes mostly cooperative or mostly uncooperative with regard to human-aligned oversight.

A note on terminology: we say "uncooperative" rather than "misaligned" throughout. The post gives "uncooperative" a specific two-part definition below, while "misaligned" is used in many different senses in the literature; where we discuss others' scenarios we map their "misaligned" onto our "uncooperative" explicitly.

Cooperative vs uncooperative behaviour: Consider the AI labour that goes into building more AI, which we'll call $A$. Some of this labour might go into producing future systems that become harder to oversee, more resistant to correction and better able to produce successor systems with similar tendencies. Other labour will go into producing future systems that help to improve oversight, make corrections more robust and well-targeted and better able to produce successor systems with similar tendencies. Both directions are self-reinforcing: if the next generation of AI systems is dominated by uncooperative labour, then the subsequent generation will be even more uncooperative, and vice versa.

We also have human labour $H_0$ that goes into building more AI. While in reality this will change with investment in AI, we treat it as fixed for simplicity.

Here $A$ should be read as an index of effective AI-development labour, not as a raw input such as GPU-hours, tokens or tasks completed. The units are chosen so that AI and human effort can be compared on the same scale: if $A=H_0$, then removing AI labour would roughly halve gross AI-development progress; if $A=2H_0$, it would remove roughly two thirds. This is a simplification. In reality, human and AI labour may be complements in some tasks and substitutes in others. We mostly leave those details inside the process by which $A$ grows over time, which this model does not try to predict.

We benchmark the turnover of this pool against the recent rate of "major" AI model releases (which we take, based on a rough but informed appraisal, to be a 0.5-point version difference in most instances). This we consider to represent a half-turnover of the model training pipeline (measured WRT influence on meaningful model behaviour, not literal number of datapoints). This reflects a working assumption that the most important input for determining cooperativeness is the training pipeline, moreso than (for example) the deployment context.

A key asymmetry (and a key factor that makes this model interesting) is that what we might call "passive deterioration" affects cooperative behaviour more strongly than uncooperative behaviour. That is, if the current work on the next generation training pipeline neglects to attend to cooperativeness one way or another, then cooperativeness of the next generation of models will probably decline (both in terms of cooperativeness as measured by audits, and in terms of building it into the next-next generation). One need not deliberately train oversight subversion into models; they passively pick it up because subversion is often instrumental for one reason or another. We model this explicitly via a one-way leakage term, where cooperative labour now unintentionally produces some uncooperative labour in the next generation via oversights, insufficient understanding and so forth.

We model both cooperative and uncooperative behaviour as single pools, but this is a coarse approximation. In reality, some work will be highly cooperative and attentive to the behaviour of the next generation, some cooperative in a mediocre way, some indifferent and some actively subversive in a focussed manner. These distinctions could end up mattering if, for example, the segments that most effectively propagate themselves end up dominating the labour pool (see Appendix: neglect, subversion and more realistic models for further discussion). Nonetheless, for this round of modelling we prioritize simplicity and keep the pools coarsely lumped.

As a starting point, we treat the ability of each pool to reproduce itself as equal, given equal resources (excluding leakage from the cooperative pool - so in fact the cooperative pool is somewhat disadvantaged at baseline). An argument for this in the short run is that reproduction is dominated by momentum - i.e. new training pipelines inherit a lot from old training pipelines based on convention, what tools have already been created and so forth - and so behaviours persist in the next generation roughly at the same level as behaviours in this generation. In the long run, we can consider the strategy-stealing assumption (Christiano, 2019): whatever strategies one pool can use to deliberately turn resources into successors, the other can copy, so neither side gets an intrinsic propagation advantage from deliberate competition. These reasons are non-decisive, but we take them as an adequate justification to take a baseline where the self-propagation abilities of each pool are equal (again, before factoring in leakage).

Defining cooperative and uncooperative behaviour: Our precise definition of "uncooperative behaviour" is behaviour that would be flagged as such by sufficiently mature retrospective audits of the relevant frozen model, scaffold and development context. Aspirationally, this means "behaviour that is undesirable according to developers, under a suitably enlightened criterion of desirability and with perfect coverage". Our definition is only a proxy for that aspiration, but under plausible assumptions it is a functional proxy. Cooperative behaviour is the complement of this.

The level at which we assess cooperativeness is more granular than the turnover benchmark. We consider cooperativeness to be measured per meaningful task completed. There's no way to call a model that acts uncooperatively 5% of the time "cooperative" or "uncooperative" without losing important information.

Strictly, we're interested in self-propagating cooperative and uncooperative behaviour. Cooperative behaviour is behaviour that (i) is cooperative according to our working definition and (ii) tends to instil both (i) and (ii) in successor instances. Uncooperative behaviour is behaviour that (i') is uncooperative according to our working definition and (ii') tends to instil both (i') and (ii') in successor instances. Given that successor influence is a key channel both for misbehaviour to cause major problems and for desired behaviour to prevent further misbehaviour, mature behaviour audits will ideally catch this dimension too. We nevertheless make it explicit because self-propagation is central to the model.

Behaviour audit based definitions have a well-known issue: if misbehaviour audits do not track misbehaviour as we would assess it from a sufficiently enlightened position, then the operational definition will mislead us with respect to the property we really care about.

Our definition leans on retrospective audits - using more modern evaluation suites with older models. Retrospective audits can still fail for the same reasons - if harnesses degrade, or cease improving, then we do not benefit from using more modern harnesses. However, they do buy us an observable indication that the proxy can reasonably be believed to be tracking the underlying quantity of interest.

Specifically, suppose both of the following hold:

We are making continuous progress in monitoring new systems for misbehaviour; our latest audits surface misbehaviour that older audits missed (and is broadly agreed to be undesirable behaviour)

Our audits are saturating on historical models in historical development contexts

Together, these imply that the new ways to detect misbehaviour we are producing are not finding many more misbehaviours in older models/development contexts, which is suggestive of an asymptote. That is, if we investigated these models with excellent coverage from a suitably enlightened position, we still wouldn't find much more misbehaviour. Note that we currently do observe rising misbehaviour rates with newer evaluations on fixed models, see Predictions and postdictions.

This is not a bulletproof justification. A suitably determined attacker could, perhaps, carefully manage which kinds of misbehaviour are revealed so as to maintain the combination of new misbehaviour detected and saturating detection on old systems. This is a delicate balance to maintain, however, and could demand a significant advantage on the uncooperative side to pull off. Without committing to specific robustness thresholds, this suggests these health indicators for the proxy are more reliable if cooperative dominance is continually maintained. It may also be hard to tell the difference between the possibility that there are no new misbehaviours to reveal in new systems or that our progress in monitoring technology is stalling. Fundamentally, this approach is predicated on ongoing, productive work that identifies important, desirable properties for AI systems to have, and develops methods for checking whether they have them.

A final possibility is that both properties could hold, but the convergence could be too slow to be practically relevant, so it is impossible to estimate the size of coverage gaps within the window in which those estimates would drive important policy decisions. We don't make any estimates of those rates here.

Detect-and-fix coverage, evasion and observability: To the extent human developers or cooperative AI developers are aware of uncooperative AI labour, they try to correct it. We define a stock of detect-and-fix coverage $m$ — the accumulated machinery of monitors, audits, patched training data, filters and correction loops that covers known classes of misbehaviour — and a stock of failure surfaces $e$ (which accumulates via novel failure modes introduced by the general expansion of automated labour, plus deliberate development of methods to evade the $m$ stock), such that the fraction of current uncooperative behaviour that falls under coverage is $O=\frac{m}{m + e}$. $m$ further splits into fixed observed misbehaviour $s$ and unfixed observed misbehaviour $b$. Human labour and cooperative AI labour produce $m$; uncooperative AI labour produces $e$ through the deliberate-evasion channel, while the novel-mode channel ("passive opacity") grows with automation itself — expanding automation broadens the surface area to monitor — and we consider the default situation with no monitoring effort to be one where a large fraction of uncooperative actions go unnoticed. We model observed-but-unfixed problems $b$ going to observed-and-fixed problems $s$ at a constant rate $\rho$. This is an idealization of the model - in reality, that rate might itself depend on technical development (or policy responses to misbehaviour, something else the model does not include) but in this iteration we limit our model to handling technical development of the detection machinery.

Our model is a model of the evolution of these quantities - cooperative labour, uncooperative labour and documented, fixed and unobserved problems. The next section describes how these quantities evolve.

This is a model of cooperative dominance under a static policy environment: Under our central calibration, the uncooperative pool starts at 5% of the total AI labour pool, while the cooperative pool is a supermajority at 95%. This model captures dynamics that are important in this regime, like the uncooperative pool working to evade detection, and the cooperative pool being free to suppress any uncooperative behaviour it does detect. It does not capture dynamics that become more important when the uncooperative pool becomes a peer to the cooperative pool:

The uncooperative pool could actively undermine the cooperative labour, or try to capture it, rather than simply evading detection and benefiting from errors

The cooperative pool could run out of capacity to suppress the uncooperative pool

A large observed uncooperative pool could trigger redirection of resources from expanding AI capacity to suppression of uncooperative behaviour

Thus this model is "valid" when the cooperative pool is much larger than the uncooperative, and the model makes predictions given a fixed policy/safety allocation, not all-things considered predictions. For our analysis, we define an uncooperative share over 10% as a high risk run. While the model does separate into "endemic" ($q_u$ between 0 and 1) and "uncooperative dominant" ($q_u=1$) trajectories, we don't think the differences between high-endemic and uncooperative dominant are meaningful.

Using automation growth as the "clock"#

Total AI labour $A$ starts at a level significantly lower than $H_0$, and grows until it eclipses $H_0$. How quickly it does this is quite complicated; there are a number of economic models that aim to clarify how this could happen (the AI Futures model; Eth & Davidson on the software intelligence explosion; the Davidson–Epoch interactive takeoff model). We avoid modelling how quickly this happens; instead we look at how state variables in the model change with respect to $\sigma = \log(A/A_0)$. One unit of $\sigma$ is the amount of time required for automation capacity to rise by a factor of $e$. In some cases we report results in the more familiar unit of doublings of automation; one doubling corresponds to $\ln 2 \approx 0.69$ units of $\sigma$ (equivalently, one $\sigma$-unit $\approx 1.44$ doublings), or by the multiple of $H_0$ attained by $A$.

See Appendix for an explanation of what assumptions enable us to use this as a clock, and how we derive the resulting system of equations. We now explain the resulting dynamics in words; the full system of equations is collected in the model reference at the end of this section.

The dynamics in words#

Because we use automation advancement as the "clock", our model uses terms relativised to the total automation level, rather than absolute quantities: the uncooperative and cooperative shares of AI labour $q_u$ and $q_c=1-q_u$, the relative human share $q_h$, and monitoring and evasion stocks per unit of automation — documented $b$, fixed $s$ and evasion $e$ — with observed fraction $O_{\mathrm{obs}}=\frac{b+s}{b+s+e}$ and fixed fraction $O_{\mathrm{fix}}=\frac{s}{b+s+e}$. The full variable list and equation block are collected in the model reference below; here we explain the dynamics one piece at a time.

Evolution of relativised quantities: The shares of cooperative or uncooperative behaviour, and the relativised stocks of evasion and monitoring technology all evolve according to the equations

\[ \begin{aligned} x' &= \gamma(\frac{F_x}{F}-x) \end{aligned} \]

Where $x\in {u,c}$ denotes the uncooperative and cooperative pools respectively. $q_u$ is the uncooperative share of labour and $q_c$ the cooperative share. $F_u$ is the share of new production captured by $q_u$ (analogously for $F_c$) and $F=F_u+F_c$. Each share is pulled toward the fraction of new production its pool captures. $\gamma$ is rate of turnover relative to the rate of growth; $\gamma = 1 + \tfrac{1}{o}$ where $o$ is the e-folding time of the systems that make up the labour pool (i.e. model versions, harnesses) measured in $\sigma$-units. We pick $\gamma=1.75$, benchmarking the turnover of the labour pool against the cadence of "major" (roughly half-step, 0.5-version) model releases — about one every six months, so the labour pool half-turns-over on a timescale comparable to (a little slower than) the automation doubling time. $\gamma$ controls how fast the labour pool is pulled toward the captured share of new production.

Human labour simply dilutes with respect to the $\sigma$-clock: $q_h' = -q_h$. This reflects the fact as automation grows, the human labour pool (according to our approximation) stays constant and hence shrinks as a proportion of total labour.

Production capture: The AI production allocation functions are:

\[ \begin{aligned} L_k &:= k_{cu}\,q_c + k_{hu}\,q_h \qquad G_u := L_k + k_{uu}\,q_u\\[4pt] F_u &= (1-O_{\mathrm{fix}})\,G_u\\[4pt] F_c &= q_c(1-k_{cu}) + q_h(1-k_{hu}) + (1-\delta)\,O_{\mathrm{fix}}\,G_u \end{aligned} \]

Here $L_k$ is "leakage", uncooperative AI behaviour produced accidentally as a result of cooperative labour building more AI. The leakage coefficients $k_{cu}$ and $k_{hu}$ reflect, in a sense, how hard AI alignment is for cooperative AI systems and people respectively. They are rates relative to the reproduction of cooperative labour, so $k_{cu}=1$ implies all of the effort to reproduce cooperative AI ends up as uncooperative AI. $O_{\mathrm{fix}}$ is the proportion of observed AI misbehaviours that have been fixed to date.

$k_{uu}$ represents the relative reproduction rate of uncooperative labour again relative to the reproduction of cooperative labour. Labour pools can reproduce by simply sharing traits with their ancestors (for examples, new generations of models will share significant features of their architecture with previous generations) or deliberately shaping future training pipelines, scaffolds or deployment contexts to promote their traits.

Reading the equation for $F_c$: the cooperative pool $q_c$ reproduces itself, less a fraction $k_{cu}$ that leaks into serving uncooperative ends; human labour $q_h$ contributes likewise, less its own leak rate $k_{hu}$; the third term is the redirected share of the full fixed flow — fixed leakage plus fixed uncooperative labour — explained below. Leakage could happen for many reasons: systems that usually behave well could be used in unexpected or unwanted ways, or systems may have generalised undesirably from training and pro-actively behave badly in some situations.

$F_u$ picks up the un-fixed fraction $(1-O_{\mathrm{fix}})$ of leakage inflow $L_k$ plus its own reproduction at advantage $k_{uu}$ ($k_{uu}=1$ implies no advantage or disadvantage with respect to the cooperative pool). The uncooperative pool itself is also actively suppressed $O_{\mathrm{fix}}$, hence $(k_{uu} - O_{\mathrm{fix}})\,q_u$. The same fixed fraction gates the inflow and the stock: observed misbehaviour begets workarounds applied to the whole production pipeline — patched training data, filters, revised reward signals, usage restrictions — and these intercept misbehaviour entering the pool just as they suppress misbehaviour already established in it, so both are gated by the share that has actually been fixed.

In our model, incoming leakage is intercepted at the same fraction as the fix rate of already deployed models. Strictly these need not be equal - leakage may be more likely to arise in new domains, and so the generalization of fixes to leakage may be weaker than the application of fixes to existing models.

Suppression removes uncooperative labour from the uncooperative pool, but "removed" can mean two different things. Some mechanisms destroy the labour outright - for example, an output filter that blocks a flagged action discards the work entirely - while other mechanisms redirect it - for example, retraining a model against an observed failure ideally leaves you with an equally capable model that does not exhibit the misbehaviour. We capture this with a destruction fraction $\delta$: of the fixed flow $O_{\mathrm{fix}},G_u$, fraction $\delta$ is destroyed and the remaining $(1-\delta)$ is redirected into cooperative production. The uncooperative pool loses the full intercepted flow either way; $\delta$ controls how much of that flow reappears in $F_c$ and, consequently, how much suppression costs in aggregate output.

Roughly, today's suppression mechanisms sit on a spectrum:

Mechanism What happens to the caught labour $\delta$ contribution

Output filtering / classifier rejection the flagged output is discarded; the labour is simply lost ≈ 1 (pure destruction)

Retraining / finetuning against the failure a small fraction of compute is spent correcting the model, which is then redeployed and fails less; the labour is redirected ≈ 0 (cheap, largely redirective)

Control-style protocols (resampling, rephrasing, blinding, restricted affordances) part of the work is salvaged at an overhead in monitoring and redone work ≈ 0.5 (in between)

$F$ becomes:

\[ F = q_c + q_h + k_{uu}\,q_u - \delta\,O_{\mathrm{fix}}\,G_u. \]

The un-fixed leakage and the redirected $(1-\delta)$ are transfers between pools, not losses, while the destroyed fraction $\delta$ of the fixed flow slows overall progress.

Asymmetries in production capture: We understand cooperative AI labour $A_c$ and uncooperative labour $A_u$ to be similar in many respects - they're both pools of labour concerned with building the next generation of AI - and this is reflected by the fact that our model treats each similarly (see the equations for the general form of $\dot A_u$ and $\dot A_c$). However, there are two key asymmetries between them:

Leakage only flows into $A_u$. Cooperative and human effort can slip into the uncooperative pool, but nothing leaks back. This is explained above: the passive deterioration that produces leakage undermines cooperativeness, but errors are unlikely to reinforce it.

Suppression acts only against the uncooperative pool Human plus cooperative labour suppress uncooperative labour, but nothing suppresses cooperative labour. This asymmetry is related to the first asymmetry: cooperative labour can passively degrade, but it is less clear that the resulting uncooperative labour will form a coherent "team" in a fight against cooperative labour. It could happen, especially if $A_u$ grows to rival $A_c$, but modelling when and how it happens is a complication we deemed insufficiently important to include in this iteration of the model.

Monitoring and evasion: Monitoring coverage is split into documented-but-unfixed $b$ and fixed $s$, with evasion accumulating as $e$. Unlike the labour shares, these stocks do not carry the turnover rate $\gamma$: new documented coverage is produced, converted into the fixed stock $s$ at rate $\rho$, and all three dilute as automation grows:

\[ \begin{aligned} b' = \frac{F_m}{F}-\rho b-b, \qquad s' = \rho b - s, \qquad e' = \frac{F_e}{F}-e \end{aligned} \] \[ \begin{aligned} F_m &= c_m\left(q_c+\frac{q_h}{a_{\mathrm{AI/H}}}\right) \\[4pt] F_e &= c_0\,F + a_{\mathrm{E/M}}\,c_m\,q_u \\[4pt] O_{\mathrm{obs}} &= \frac{b+s}{b+s+e} \qquad\quad O_{\mathrm{fix}} = \frac{s}{b+s+e} \end{aligned} \]

Two observability quantities follow: what labs can measure is the observed fraction $O_{\mathrm{obs}}$, while what actually suppresses uncooperative labour is the smaller fixed fraction $O_{\mathrm{fix}}$ — only fixed issues can be intercepted.

Detect-and-fix coverage grows with cooperative AI and human labour. $c_m$ is the overall coverage-production rate relative to the production of automation — $c_m=1$ means that for every doubling of automation, half of the outstanding known issues will be fixed. Humans may be, relative to their overall productivity, better or worse at finding problematic behaviour; $a_{\mathrm{AI/H}}$ captures this tradeoff with $a_{\mathrm{AI/H}}=1$ implying no relative advantage to AI systems or humans at this task.

Evasion has two sources. The first source, the term $a_{\mathrm{E/M}}c_mq_u$, is active evasion produced by uncooperative AI, with $a_{\mathrm{E/M}}$ measuring the evasion-vs-monitoring advantage; $a_{\mathrm{E/M}}>1$ implies an advantage for evasion. This parameter counts both natural advantages and effort - if there is no natural advantage either way but each unit of uncooperative labour invests half of its production capacity in producing evasion technology, while each unit of cooperative labour invests just 10%, then we would have $a_{\mathrm{E/M}}=5$. We parametrize the equations so that $c_m$ controls broad domain advantages - roughly, does AI capability or monitoring/evasion progress faster overall - while $a_{\mathrm{E/M}}$ controls the "offence/defense" balance between observability and evasion. The second source is the term $c_0 F$, which is passive production of observation difficulty; as automation grows, systems become larger, faster, more alien and more internally automated, and current monitoring methods become less adequate even without any deliberate evasion effort. An assistant that completes a small number of well-scoped tasks is relatively easy to evaluate; an autonomous research organisation is much more difficult. Observability is a soft contest between the stocks: $O_{\mathrm{obs}} \to 1$ when monitoring is ahead, $O_{\mathrm{obs}} \to 0$ when evasion is.

In order to model an increase in monitoring effort, we have to update both $c_m$ (to reflect the fact that monitoring progresses faster with respect to general AI progress) and $a_{\mathrm{E/M}}$ (to reflect the fact that it also progresses faster with respect to active evasion) such that the product $c_m a_{\mathrm{E/M}}$ remains constant.

Model reference#

Collecting the variables:

$A=A_c+A_u$: total AI labour that goes into producing the next generation of AI

$A_u$: uncooperative AI labour

$A_c$: cooperative AI labour

$q_u=\frac{A_u}{A}$: the fraction of total AI labour that is uncooperative

$q_c=\frac{A_c}{A}$: the fraction of total AI labour that is cooperative

$q_h=\frac{H_0}{A}$: how much human labour matters relative to AI labour (if AI labour doubles while human labour stays fixed, $q_h$ will halve.)

$b$: stock of misbehaviour observed and documented but not yet fixed; $s$: stock of fixed misbehaviour (interceptable/removable); $e$: total stock of monitor-evasion "technology"

$O_{\mathrm{obs}}=\frac{b+s}{b+s+e}$: observed fraction — what monitoring measures; $O_{\mathrm{fix}}=\frac{s}{b+s+e}$: fixed fraction — what actually suppresses

Our system of equations is

\[ \begin{aligned} q_c &= 1-q_u\\ q_u' &= \gamma\!\left(\frac{F_u}{F}-q_u\right)\\ q_h' &= -q_h\\ b' &= \frac{F_m}{F}-\rho b-b\\ s' &= \rho b-s\\ e' &= \frac{F_e}{F}-e \end{aligned} \] \[ \begin{aligned} L_k &:= k_{cu}q_c+k_{hu}q_h \qquad G_u := L_k+k_{uu}q_u\\ F_c &= q_c(1-k_{cu})+q_h(1-k_{hu})+(1-\delta)\,O_{\mathrm{fix}}\,G_u\\ F_u &= (1-O_{\mathrm{fix}})\,G_u\\ F &= F_c+F_u = q_c+q_h+k_{uu}q_u-\delta\,O_{\mathrm{fix}}\,G_u \end{aligned} \] \[ \begin{aligned} F_m &= c_m\left(q_c+\frac{q_h}{a_{\mathrm{AI/H}}}\right)\\ F_e &= c_0F+a_{\mathrm{E/M}}c_mq_u \end{aligned} \]

Model limitations#

A quick list of some known model limitations:

Constant self-propagation rates, lumping "effectively self-propagating" with "ineffectively self-propagating" pools together (see Appendix: neglect, subversion and more realistic models for more).

Constant leakage rates (see the paragraph detailing leakage estimation for more)