AI #176 Part 2: Plan B

Zvi Mowshowitz

Jul 10, 2026

This is part 2 of the weekly, broadly covering speculation, rhetoric and policy, along with alignment research.

This does not cover the release of GPT-5.6-Sol. As always, I will be taking a few days to digest what the new model has to offer and to allow others to try it and react. I will cover Sol and its capabilities early next week. I covered the GPT-5.6 system card back on June 28.

This also does not cover the release of Plan A, the follow-up to AI 2027. This new scenario is a positive vision of what its authors think we should do going forwards.

I do not endorse all of the recommendations or predictions of Plan A, but I do endorse reading Plan A and taking it seriously. Scott Alexander, one of those who worked on it, writes an introduction and justification here. I will have full coverage soon.

Table of Contents

Quiet Speculations. Will our AI regulations be ad hoc indefinitely?

The Goalposts Are Dyson Spheres. This might take a little longer.

People Just Say Things.

Three Pills. Unpilled, AI, AGI, ASI.

The Quest for Sane Regulations. You’ve got to have guardrails.

OpenAI National Security Principles. Good ideas, but how to enforce them?

Greetings From The Department Of War. New primary documents.

Chip City. Nvidia keeps lying right to the government’s face. Also yours.

Open Weight Models Are Unsafe And Nothing Can Fix This. It is time.

Their AI Propaganda Bots. Oh, I’m sorry. This is abuse.

Rhetorical Innovation. Security mindset.

You Learn. But did it really count?

You May Be Tan And Thin And Rich But You’re a Tool. I, on the other hand…

Train Those Thoughts. Yay finding the error, boo the actual error. So confusing.

Train Out Those Thoughts. GRAM as a new training technique.

My Own Private Idaho. You have nothing to hide. Privacy is still worthwhile.

Aligning a Smarter Than Human Intelligence is Difficult. How’m I doin?

No Space Like J-Space. Let’s see, what do we have here? Oh.

Cooperative Alignments. Claims about Sonnet 5.

The Lighter Side. You have not been a good user.

Quiet Speculations

Outgoing White House advisor Sriram Krishnan says Donald Trump will never support a formal licensing regime for AI.

Joe Miller: Donald Trump will not establish a formal licensing regime for AI, the president’s departing AI adviser has said, even as the White House wields emergency powers to stall the most advanced models.

… “This administration, [the] president, from day one has been against burdensome, onerous, bureaucratic red tape,” he added. “We are not in the business of picking winners and losers.”

If true, this means they will instead continue a fully ad hoc regime (e.g. ‘there will be guardrails’), where winners and losers are chosen according to the whims of the administration. Presumably they think that approach has its advantages.

Despite this, Sriram’s statement does not end with And That’s Terrible.

Joe Miller: Krishnan said setting up a centralised agency requiring “a team of lawyers before you can get a model out” would put “sand in the gears” of the AI revolution. “That is never, never going to happen under President Trump,” he said.

Instead, you will have to go through a fully opaque, arbitrary ad hoc process that involves the whims of power. Much better, you see. Krishnan also supports extorting equity from major AI companies, as part of this regime.

Joe Miller: Krishnan warned that if cutting-edge AI tools were held back by the government for several weeks, “that would probably be bad for American innovation”.

Some of us are worried about much worse outcomes than a few weeks of delay. GPT-5.6 is available as of yesterday, so the delay was annoying but not an epic deal.

Joe Miller: Asked whether a future Democratic government could use the Trump administration’s unilateral use of export controls as a pretext to stall the rollout of AI, Krishnan said: “I don’t think about future governments. I think about this government and this moment in time.”

I think we may have found part of the problem, right there.

Tyler Cowen, never stop Tyler Cowening, as he speculates on how AI and the fertility crisis will intersect in ways only he could, holding the world constant so he can center things he finds interesting at the expense of many more important questions. He imagines a world of depopulated cities where people talk to each other and focus on looking and being good, unique and interesting to compete with AI interactions. The first comment claims Tyler is biased in favor of predicting social change, but actually this is Tyler predicting remarkably little social change.

Carlo Cordasco claims that transformative innovations have their costs clear right away, but their benefits only become clear over time. This seems not right, even historically. Instead, Carlo tells the story of various warnings about technology downsides (e.g. Plato on writing) that did not pan out. Such wrong warnings are common, but also often the biggest downsides are missed. That doesn’t mean we are sad about the printing press or radio, or social media and smartphones, or agriculture, or birth control, or burning fossil fuels, and so on, but no we do not see the costs up front. I think it’s better to say that transformational technological impacts are historically hard to predict.

The Goalposts Are Dyson Spheres

Once again we get another entry into ‘oh but you didn’t realize that physical actions take nonzero amounts of time to figure out and test, now did you?’

Epoch AI: Will we get Dyson Spheres a few years after automating AI R&D? Most AI futurism debates answer this by looking at AI capabilities, but miss half the picture: how intrinsically hard it is to build futuristic tech in the first place.

New essay by @datagenproc and @ansonwhho .

Once AI research is automated, AI could rapidly surpass human experts across almost all cognitive domains.

However, that doesn’t tell us how long after that we’d have “sci-fi” tech like dyson spheres, nanotech, near light-speed travel, brain uploading, or interstellar probes.

That doesn’t mean that Dyson Spheres are impossible within a few years of automating AI R&D. It just means that forecasts of speculative technologies should be sensitive to the specifics of how hard they are to build.

They suggest that you think about, given assumptions about how capable is your AI, you look at how hard a specific technology is to build and estimate how long it would take.

Their example is a drop-in worker replacement that runs on an H100.

I mean, yes, sure, that is a good exercise to do, but as framed it presumes the AI is a fixed level of capable. The whole point of recursive self-improvement (RSI) is that the AI gets a lot better.

If you wanted to use H100 drop-in worker replacements to build a Dyson sphere as quickly as possible, any gamer would know that - aside from maybe some long lead time experiments or other groundwork you need to start well in advance - that you don’t start building or even researching the Dyson sphere right away. You keep running up the intelligence tech tree first, well past that point, and only then try to build the sphere. That’s the point of RSI.

They give their third objection, that it’s hard to model superintelligence and that the AIs will start doing things we didn’t anticipate and perhaps cannot imagine, short shrift. It’s clear that the idea is, focus on what we know non-ASI AIs will be able to do, using methods we know about, and then base our plans largely on that, knowing that maybe ASIs could do more.

JS Denain: For example, suppose you had a billion AIs that were each at least as good as top human experts at virtually all cognitive tasks.

If you could show that these AIs could build molecular nanotechnology within a few years, then surely a billion much smarter AIs could too, even if you don’t know how to model them. And if you can show that these AIs probably can’t do this, then at least this helps us identify cruxes, focusing debates on more concrete AI capabilities.

That’s a lower bound, and I would say in this case not such a useful one, and it leads to reinforcing of the idea that the future consists only of implementations of things we already know we can do.

But yes, sure, this kind of thing is sufficiently valuable to decision making that someone should do the estimation work here. It is especially useful because we do not know that we will get superintelligence, and we would like to know what we could still do without it, if either it is too hard to do or we find a way to choose not to build superintelligence for a while.

Or alternatively ask Fable or Sol to do the estimations for you.

People Just Say Things

Fact checker Megan Herbst gets the facts wrong in Wired, claims AI fact checking cannot be trusted based exclusively on AI overviews. If you want to know exactly what the studies found and what it actually means Dan Williams asked Fable for you. Megan then concludes ‘AI is wrong about half the time’ and that all fact checking must be human. This is how a lot of our intellectual elite is handling AI.

Three Pills

The three are: AI pill, AGI pill, ASI pill.

Roughly:

AI pill: AI can do the cognitive things it can already do.

AGI pill: AI will in the future do more of the cognitive things.

ASI pill: AI will in the future do all of the cognitive things.

To count as pilled you must also grapple with the implications. If you then treat this as an isolated curiosity and assume nothing much changes, it doesn’t count.

If you don’t take the ASI pill, you are responding to the wrong long term future.

If you don’t take at least the AGI pill, your AI policy won’t make sense now.

If you don’t take at least the AI pill, you might be most of the government.

One must understand that those who have strongly influenced White House policy flat out refuse to acknowledge the capabilities of existing closed frontier models, and have been playing that game for years.

Dean W. Ball: Basically I think that, back in 2023 or so, the “consistently wrong about AI” VC and SaaS community was operating under the assumption that AI’s trajectory would mean model capabilities peaking around GPT 5.5/Opus 4.8 capabilities somewhere around 2030, plus robots.

And if that was your assumption, I can totally understand why you think everything commodifies/frontier AI isn’t a legitimate business model, etc.

That is a nice world to believe in! In the real world, however, that community has been wildly wrong for three years, and I would expect them to continue being wrong for more years to come.

They may not be wrong forever! Things eventually commodify. But people have been saying “the models are good enough” since GPT-4, and it’s been untrue. I suspect that will continue to be the case because I think that we remain in the earlier stages of the AI industry, and along the steep part of the trajectory.

More broadly: the notion of “good enough” should gross you out, a little bit. The economy of the future will be about heavy-tailed excellence, not middle-of-the-bell-curve, loser-premise, “good enough”-ness.

Daniel Eth (AI Safety): Also explains why they’ve taken this insane, radically anti-regulatory stance towards frontier AI risk

Dean W. Ball: yep. increasingly their discourse feels like it requires disbelieving your eyes, imagining eg a political economy where open-source models profoundly more capable than Mythos/fable are encouraged by governments just as governments begin hardcore regulation.

If you take recursive self-improvement (RSI) and superintelligence (ASI) seriously, this changes everything. We don’t know what level of seriously we need to take them how fast, which contrasts this pill with the first two pills.

prinz: In the age of RSI, the claim that models will commoditize looks increasingly dubious. The gap between the frontier and the second tier is already huge (much larger than the benchmarks suggest), is cl

[truncated for AI cost control]