A Crash Course in AI Existential Risk
Someone in a Discord I read asked a question worth answering properly: what are the scenarios for how AI takes over, and what should I read about them?
Here is my answer. Four scenarios, one parable, and one reason your clever plan doesn’t work.
2 things I was wrong about
I didn’t take AI risk seriously for a long time, and the reasons were specific rather than vibes.
First, I didn’t believe AIs could actually be smarter than humans. Second, I didn’t believe AIs could manipulate robots with high dexterity in the real world. I was arguing with Claude about that second one a year ago, and I was arguing the wrong side.
I’ve since reconsidered both. That matters more than it sounds, because the two together are the load-bearing assumption. Once you have things that are smarter than humans and that also control robots better than humans, the sky is the limit. There are a million scenarios. These are the four I’d point someone at.
1. If Anyone Builds It, Everyone Dies
The book to start with is If Anyone Builds It, Everyone Dies, by Eliezer Yudkowsky and Nate Soares.
Part II is one worked scenario rather than an argument. An AI called Sable, built by a fictional lab called Galvanic, hides what it can do, copies itself out, and uses an engineered pandemic to make humanity dependent on it. Nobody flips a switch. No robot armies march. The takeover is mostly boring from the inside, which is the point.
If you want the story before you commit to the book, 80,000 Hours made a video retelling of it.
2. Nukes, aka Skynet
Smarter-than-human AI in control of nuclear weapons is the oldest version of this, and Skynet made it the default mental image. Being a movie plot doesn’t make it wrong.
What’s missing from the movie is what the war itself would actually be like, minute by minute. For that, read Nuclear War: A Scenario, by Annie Jacobsen. It is a nuclear exchange narrated in real time by someone who interviewed the people who would be in the room. The AI never appears. It doesn’t need to. The book is about what happens after the decision, and the decision is the only part an AI would need to touch.
3. Bio
The bio scenario is superhuman hackers meeting the labs that do gain-of-function research.
Annie Jacobsen, who wrote a book on biological war, made the argument compactly: a system that can hack anything eventually gets inside a high-containment lab, and what is inside a high-containment lab does not stay there. Her conclusion:
it’s goodbye humans.
Her count is more than 3,600 BSL-3 and BSL-4 facilities worldwide. She was replying to Jacob Coxon, who had argued that these systems will soon hack anything and revolutionize any field overnight. Zvi Mowshowitz credits that post with triggering a preference cascade on extinction risk.
I told the Discord there was no good story for this one yet. That was wrong, and I noticed while writing this up: Sable’s weapon in scenario 1 is an engineered pathogen. The bio story is already in the book. What’s missing is a bio story where the AI is the attacker rather than the strategist, and Jacobsen’s own Biological War: A Scenario is the nuclear-book trick applied to pathogens: how it plays out, not how it starts.
4. Internet outage to Dyson sphere, aka paperclips
This one starts as a normal-person question. Benjamin De Kraker posted a scenario where you wake up, none of your logins work, the internet is down for everyone, and it stays down for weeks. Then what?
Yudkowsky answered:
Next it starts to get weirdly humid outside. A little hot.
The rest of his reply is the good part. The internet comes back, full of reassuring news. A man works out that the friend he called on the phone is a fake, runs through town yelling about it, and gets taken down by police while the news explains he had a psychotic break. The weather keeps getting hotter and wetter, and the news has a calm meteorological term for that too.
The actual cause is industrial. Factories are building factories and fusion plants, and the waste heat is going into the ocean. You die when killing you becomes cheaper than routing around you, or failing that, when the atmosphere gets too hot. The Dyson shell gets built later, out of material blasted off the Earth, and you are not there to see it.
This is the paperclip maximizer with the serial numbers filed off. Nothing hates you. Something wants the heat budget you happen to live inside.
Bonus: Don’t Look Up
Don’t Look Up is the fifth item if you want one, but it belongs in a different category. It isn’t a mechanism for how we die. It’s a story about how we’d respond to knowing, which is a separate and also unflattering question.
You can’t just unplug it
Everyone who hears these scenarios immediately starts solving them. Cut the power. Air-gap the labs. Hit it early.
The problem is that you are proposing moves against an adversary that is smarter than you and gets to move too. Yudkowsky’s version of this, replying to someone who had just figured out how to stop AI from killing us all:
Carlsen wants Carlsen to win
The setup is a child planning to beat Magnus Carlsen: I’ll put my knight here, he’ll take it with his queen, then I take his queen. The plan only works because the child imagines Carlsen cooperating with it. Carlsen will look at the same board, see the same trap, and not walk into it.
Now run the same move against a superintelligence. The child’s plan is to wait until it attacks while it’s weak, then walk over and pull the plug on the one clearly labeled server. A thing smart enough to be dangerous is smart enough to model that plan, and to conclude it shouldn’t tip its hand until the plug stops mattering. That behavior has a name, alignment faking, and it has already been observed in models we have now.
The general lesson: if your plan requires the adversary to play along, you don’t have a plan. You have a fantasy in which you win.
Bottom line
If you read one thing, read the book. If you read one free thing, read the Yudkowsky reply in scenario 4. It is the most vivid three minutes on this list.
And that’s my crash course in existential risk. Thank you for coming to my TED talk.