Trees are mostly made of air and a generalizable lesson for AI safety

x

Trees are mostly made of air and a generalizable lesson for AI safety  — LessWrong

2026 Top Fifty: 13%

307

One student, upon being told that most of the mass in the wood comes from  and not the dirt in the ground, said “that's very disturbing and I wonder how that could happen.” And that’s an MIT graduate.

“Trees come from mostly air” is pretty fundamental for biology because it follows from photosynthesis which is in many ways the basis for life on earth. In a sane world, every 8th grader would know that dry wood is mostly , not like minerals found in the ground or something. MIT grads probably have vast amounts of detailed knowledge of math, physics, biology, chemistry or all of the above. But they don’t know some basic facts about biology.

There is this assumption that you should first learn the foundational facts of an area of study and then move to more and more specific questions and ideas. As you move up levels of classes, the foundational stuff seems more and more basic and less and less relevant. Some stuff is continually hammered in because it’s useful background knowledge. In a perfect world, this helps you internalize the basics and learn how to reason about them to solve harder and harder problems.

There are some areas where, without much effort, this may work out. For example, you learn fractions in first or second grade but will probably understand them more deeply by the time you get to calculus because you need to know fractions to do calculus and all the classes that come before calculus. But not all foundational knowledge is like this! Knowing where trees come from doesn’t help you answer organic chemistry or evolutionary biology questions.

Being 1. foundational and being 2. useful for answering increasingly specific questions are different things. They are certainly not orthogonal but they are also not perfectly correlated. When 1 & 2 diverge, you get MIT grads who are confused about what wood is.

In AI safety, this can be a serious problem.

I have had one-on-ones or interviewed dozens of students who want a career in AI safety. There are many examples of students who are something like this: they know what alignment faking is, read LessWrong, know who Neel Nanda is, know what METR is, have done an interp project, etc. But when you ask them why they care about AI safety they don’t provide a particularly coherent answer. So I get more specific: “Why should we think AI is an existential risk?” Again, incoherent answer.

This may be because they don’t really care much about AI safety and they just like to hang out with EA/rationalist types. An alternative reason (which I think is more likely) is because AI safety basics are something you learn and don’t exercise. When you do a BlueDot reading group you hopefully learn AI safety basics. But when doing interp experiments or your first SPAR research project, you think about specific empirical questions, not the orthogonality thesis. You don’t think about the basics and you don’t internalize them and certainly cannot reason about them.

Instrumental convergence, inner alignment, reward misspecification, etc. are our “trees are made out of air” or “the Roman Empire was 27 BC to 476 AD”. But lots of people in AI safety programs or those applying for them don’t know the basic facts. They know a lot of specific things but somehow not the foundational things.

This strikes me as a genuine failure mode. A lot of focus goes into having great fellowship programs and university groups but some conceptual knowledge seems to be slipping through the cracks. I hope the next generation of AI safety researchers have all of the conceptual knowledge of earlier researchers and more.

I’ll leave you with an additional confession. 6ish years ago, before I started college, this was probably me, at least partly. I understood the basic alignment problem but understood it mostly through outer-alignment issues and didn’t fully internalize the difficulty of the problem until I started college. That was roughly 4 years ago when I moved into the UChicago dorms. From the very beginning, I was a CS major because I wanted to be an AI safety researcher. A few hours ago I turned in my last paper, and now I’m done.

My primary reflection is this: I would not have become a CS major, would not have worked so hard, would not have been so laser-focused on AI safety if I didn’t actually understand it. I would have got distracted and ended up probably on Wall Street or worse. This is because knowing a problem tells you why you should care. So if there is one reason to embrace the basics, it is that. There is so much fucking power in actually understanding something.

^

The reason I'm saying "dead tree" or "dry tree" above is that living trees can contain a lot of water mass but this analysis only focuses on the non-water parts of the tree--i.e., the hard "wood stuff."

10emanuelr

2Luc Brinkman

37Ryan Meservey

10Charbel-Raphaël

4Tom DAVID

4Charbel-Raphaël

19kbear

47ConcurrentSquared

11Arbituram

11Rachel Shu

3kqr

19Zephaniah Roe

2kbear

4Zephaniah Roe

16Davidmanheim

13Myron Hedderson

7Steven Byrnes

2Matt

4Steven Byrnes

1Matt

7Yair Halberstadt

6Ben Livengood

6Morpheus

4Karl Krueger

11Raemon

9Oliver Sourbut

6Zephaniah Roe

2Gurkenglas

2Zephaniah Roe

2Gurkenglas

3Oliver Sourbut

6Sophia Gore

3Zephaniah Roe

5Bruce Middleton

2deep_pars

2Bruce Middleton

1XelaP

4denkenberger

3Philip Niewold

3Linda Linsefors

3Zephaniah Roe

2Linda Linsefors

3Salmonus_Kim

2Zephaniah Roe

7Salmonus_Kim

1Zephaniah Roe

2Chris_Leong

2Camille B.

2Nicolas Macé

1Antariksh

1NickH

1Koshtubh Singh Parihar

1Antoine Maier

1Patrick Reichherzer

1XelaP

1Andrii Vasylenko

0Ebenezer Dukakis

307

Click to highlight new comments since: Today at 8:40 PM

^

(i.e trees)

^

Maybe inner vs outer alignment isn't the right framing (I think it is but it's a source of controversy) but I still think this is a good question because it tests how well you can reason about different kinds of failure modes and how they may interact, etc.

Moderation Log

More from Zephaniah Roe

View more

Curated and popular this week