The wrong question
These are the days of AI’s black legend, and the AI doomers own the headlines. What matters is this: they may be right about the danger. But they may be right the way a stopped clock is right twice a day: by accident, and for none of the reasons written on its dial.
The doom scenario is real enough to take seriously, and sooner rather than later. But the reasons we are told it could happen are fatally wrong.
This story explains why, down to the math, while staying accessible to readers without a technical background, with animations and plots doing the heavy lifting where needed.
The AI Doom Debate by Tribes
Before we talk about boxes, contexts, or escape probabilities, you need to know the battlefield.
The AI doom debate has split into four recognizable tribes. Three think the machine may eventually kill us. One thinks the others need to calm down.
They disagree about timelines, probabilities, motives, and how seriously to take the whole thing. But almost all of them are still staring at the same variable: How powerful will the AI become?
That is where the argument has been stuck. And you will see that it is the wrong question altogether.
Now let’s meet the tribes.
The Rapture Bros
Eliezer Yudkowsky and Nate Soares — whose book is literally titled If Anyone Builds It, Everyone Dies — together with figures such as Dan Hendrycks and Andrew Critch, are among the most visible voices warning that advanced AI could pose an existential danger. Yudkowsky in particular has argued for extremely high probabilities of catastrophe from sufficiently advanced AI, while Hendrycks has pushed for AI extinction risk to be treated as a global priority.
Position, at the most extreme end of this camp: build sufficiently advanced AI without solving control first, and catastrophe may become overwhelmingly likely. p(doom): very high, with Yudkowsky near the extreme end.
They set the date for the end of the world, then asked you to preorder the book about it.
The Nervous Godfathers
Geoffrey Hinton and Yoshua Bengio are not exactly two random guys who discovered ChatGPT a few months ago and immediately started preparing the bunker. They are recipients of the 2018 Turing Award of the 2018 Turing Award, basically the Nobel Prize of computing, and Hinton later received the real thing, the Nobel Prize in Physics, for breakthroughs that made modern deep learning possible.
So after spending decades building the thing, making it better, and collecting some of science’s biggest prizes along the way, they suddenly looked at what they had created and thought: Hang on. How exactly are we supposed to stop this thing?
Position: they built the foundations of modern AI, and now they are warning everyone not to be too confident about where the building goes next. The danger is not certain, but it is serious enough that keep scaling and hope is not a strategy.
They built the fire, won the medals for combustion, and now show up at midnight asking whether anyone remembered to install the sprinklers.