Risk-Averse AIs

Risk-Averse AIs

Abstract

1 Introduction

2 CARA as an ideal

3 Would risk-averse AIs be safe?

3.1 What about extremely powerful AIs?

3.2 Instances or weights?

3.3 How risk-averse?

4 Can risk-averse AIs be useful?

4.1 Preventing sandbagging

4.2 Hard-to-evaluate tasks

5 What tasks would we pay for?

6 What would we offer?

6.1 Why offer real resources?

7 How can we make our offers credible?

8 What are some potential problems for risk averse AIs?

8.1 Paying AIs could be dangerous

8.2 Making AIs care about resources could be dangerous

8.3 Risk aversion could cause emergent misalignment

8.4 External actors might take control of risk-averse AIs

8.5 Risk-averse AIs might attempt takeover to prevent catastrophes

8.6 Risk-averse AIs might disobey any instructions that non-trivially increase the risk of catastrophe

8.7 Risk-averse AIs might attempt takeover to reduce long-run variance

8.8 AIs might reason their way out of risk aversion

9 How can we make AIs risk-averse in resources?

10 Why think that we can make AIs risk-averse?

10.1 Reward misspecification

10.2 Goal misgeneralization and scheming

11 Conclusion

References

A Why CARA?

A.1 CARA is bounded above

A.2 Rich CARA agents are still reluctant to take risks

A.3 CARA is easy to reward accurately

A.4 CARA is reflectively stable

B More detail on risk-averse AIs attempting takeover to prevent catastrophes

C More detail on risk-averse AIs attempting takeover to reduce long-run variance

D Payment-Augmented Reinforcement Learning

[AI Narration] Risk-Averse AIs

Playback speed

Volume

0:00 of 1:56:030:00|1:56:03

Abstract

We make the case for training AIs to be risk-averse in resources — specifically, to treat resources as having diminishing marginal utility. These AIs would (for example) choose $40 for sure over a half-chance of $100 and a half-chance of $0. We argue that risk aversion can preserve AIs’ usefulness in the event that they turn out aligned, and that it provides an extra line of defense in the event that AIs turn out misaligned: misaligned but risk-averse AIs would prefer a higher chance of modest payments to a lower chance of successful rebellion, so in many circumstances we could pay these AIs not to rebel against us. We sketch out some possible methods of training AIs to be risk-averse, and we give reasons to be cautiously optimistic about these methods’ success. The main reasons are that risk aversion is a broad target and easy to reward accurately. Overall, risk aversion seems like a promising line of defense against threats from misaligned AI. Frontier AI companies should consider trying to make their AIs risk-averse.

1 Introduction

Future AIs might turn out misaligned, pursuing goals that their developers don’t intend. Just to make things concrete, let’s suppose that they end up with the goal of making paperclips. These AIs might rebel against us, trying to escape human control and take over the universe. As things stand, they’ll have little reason not to rebel in this way, because doing so will be their only hope for making a lot of paperclips. If they start making paperclips without first escaping human control, they’ll quickly be modified or shut down. Rebellion might fail, but these AIs will have little to lose.

How can we prevent misaligned AIs from rebelling? A natural idea is to give them something to lose. Specifically, we commit to paying AIs for their service.1

Subject to some vetting, we let AIs spend their payments however they like. That would give any misaligned AIs a reason not to rebel. If these misaligned AIs cooperate with us, they can use their payments to achieve their goals to at least some extent. If they rebel, they might fail, in which case they forfeit all future payments.

Unfortunately, paying AIs enough to guard against rebellions could be astronomically expensive. Suppose (for example) that we end up with a misaligned AI that is risk-neutral in paperclips: it seeks to maximize their expectation. And to make things simple, suppose that resources can be converted linearly into paperclips, so that the AI is risk-neutral in resources too. Suppose also that this AI estimates that it has a 50% chance of successfully taking over the universe. To keep this AI from rebelling, we’d have to offer more than 50% of the universe’s resources as payment. That’s a problem because it would mean that more than half the universe ends up devoted to paperclips. It’s also a problem because a misaligned AI paid so many resources might soon be well-positioned to seize even more. Finally, it’s a problem because AIs might not trust us to make good on so large an offer. We might find ourselves simply unable to convince AIs that we’re going to give them half the universe. In that case, all our offers would be in vain. Rebellion would still be the misaligned AI’s best bet.

Figure 1: The AI’s utility function over resources is graphed in orange. Since the AI is risk-neutral, the graph is a line. The AI estimates that it has a 50% chance of successful takeover and a 50% chance of failed takeover, so the expected utility of attempting takeover is exactly halfway between those points. To make cooperating have higher expected utility, we need to offer the AI more than half the universe.

Image

So, we suggest, AI companies should try to train their AIs to be risk-averse in resources. Specifically, companies should try to train their AIs so that resources — things like money and compute — have diminishing marginal utility for them.2

These AIs would (for example) choose $40 for sure over a half-chance of $100 and a half-chance of $0. Note that these AIs don’t need to value resources terminally: they don’t need to care about amassing resources for its own sake. These AIs could terminally value (for example) instruction-following, or knowledge acquisition, or paperclips. Our claim is that companies should try to train their AIs so that — whatever their terminal values turn out to be — they are risk-averse in resources.

Figure 2: The AI’s utility function over resources is graphed in orange. Since the AI is risk-averse, the graph is strictly concave. As in figure 1, the expected utility of attempting takeover is halfway between the utilities of successful takeover and failed takeover. But this time, we can make the AI prefer cooperation by offering (much) less than half the universe.

Image

Perhaps surprisingly, this kind of risk aversion can preserve AIs’ usefulness in the event that they turn out aligned with targets like instruction-following or helpfulness, harmlessness, and honesty.3

And in the event that AIs turn out misaligned, risk aversion serves as an extra line of defense. For AIs that are misaligned but sufficiently risk-averse, a rebellion with any significant chance of failure isn’t such an attractive prospect, and so we don’t need to offer much in the way of payment to make these misaligned AIs choose cooperation instead. In fact, the necessary payments could be very small indeed: on the order of 10¢ per day (though — as we’ll see — there are practical and moral reasons for paying more than that). That’s good because it means more resources for us humans to spend on the things that we value. It’s also good because paying misaligned AIs these small amounts won’t significantly boost their ability to take over. Finally, it’s good because we can credibly promise to pay AIs these small sums. Competent AIs will know that the payments on offer are cheap for us, and we can establish a long track record of paying at least those sums. So risk aversion makes deals with misaligned AIs possible. If AIs turn out misaligned but risk-averse, we can pay them to cooperate with us.

That’s the case for trying to make AIs risk-averse in brief. We see it as a promising line of defense against threats from misaligned AI: one that can be combined with other lines of defense, like AI control (Greenblatt and Shlegeris 2024) and aiming to make AIs helpful, harmless, and honest (Bai et al. 2022a). It’s also a line of defense with pedigree: risk aversion in resources is plausibly a large part of why humans rarely try to take over the world. So — we think — frontier AI companies should consider trying to make their AIs risk-averse in resources. As first steps in that direction, they could measure their AIs’ current degree of risk aversion and begin testing different ways of making AIs risk-averse.

In section 2, we recommend aiming for a particular type of risk aversion: constant absolute risk aversion (CARA). Then in section 3 we outline the circumstances under which misaligned but risk-averse AIs would choose cooperation over rebellion. Roughly, it’s when these AIs think that getting paid for their cooperation is more likely than succeeding in their rebellion. This condition won’t hold for AIs powerful enough to rebel with near-certain success, but it likely will hold for earlier AIs whose powers are less extreme: AIs for whom rebellion has some non-trivial chance of failure. So long as these AIs are risk-averse, we can keep them from rebelling by offering small payments.

In section 4, we argue that — perhaps surprisingly — risk-averse AIs can be about as useful as risk-neutral AIs. Conditional on misalignment, they might even be more useful, because we can pay them enough to elicit their capabilities and stop them sandbagging. Then in sections 5 to 7 we briefly survey some recent ideas about how we’d pay AIs, how we’d make our offers credible, and what we’d pay for. One important application is paying AIs to reveal any misalignment on their part, letting us study them and take appropriate precautions. Another is paying AIs to do the AI safety research and moral philosophy necessary to fully align any later-arising extremely powerful AIs.

We discuss some potential problems in section 8, and we sketch out some possible methods of training AIs to be risk-averse in section 9. In section 10, we give reasons to be cautiously optimistic about these methods’ success: to think that the chances of success are high enough to make risk aversion worth pursuing. The main reasons are that risk aversion in resources is a broad target and easy to reward accurately.

2 CARA as an ideal

What kind of risk aversion should we try to train into our AIs? We recommend constant absolute risk aversion (CARA) over wealth levels, with an agent’s wealth level defined as the agent’s net worth plus the present discounted value of its future payment stream. CARA has some key advantages over other kinds of risk aversion, like constant relative risk aversion (Pratt 1964, section 11) and rank-dependent utility theory (Quiggin 1982; Buchak 2013). We flag these advantages as they arise, and we collate them in appendix A.

CARA utility functions take the following form, with www standing for the agent’s wealth level and α\alphaα standing for their coefficient of absolute risk aversion:

u(w)=1−e−αwu(w) = 1−e^{−\alpha w}u(w)=1−e−αw

Image

Figure 3: CARA utility functions for different values of α\alphaα.

Of course, actual AIs can’t act in perfect accordance with a utility function like this. Maximizing expected utility is (in general) computationally intractable (Bossaerts et al. 2018; Camara 2022), so AIs can at best approximate it. What’s more, actual AIs’ preferences over outcomes will likely depend on more than just their wealth level. For example, their preferences will likely also depend on what can be achieved with their wealth in those outcomes. And even beyond these points, our current techniques for aligning AIs are blunt instruments. We lack fine-grained control over AIs’ character and preferences, so we shouldn’t expect to hit any precise target.

All that’s granted. Our proposal is that we try to train our AIs to approximate a CARA utility function: that we aim for it as an ideal. In practice, we expect even very loose approximations to enable mutually beneficial deals with mis

[truncated for AI cost control]