Be forewarned. I’m going to start on a “doomy” note. If we don’t acknowledge the problem, there is little hope of solving it. But I promise, if you read to the end, it gets better.
Geoffrey Hinton, the Nobel Laureate and Turing Award winner who helped create the algorithms that underlie much of generative AI, said there is a “10% to 20% chance [for AI / SI] to wipe us out.”1
Other researchers have similar, or even more pessimistic, views.
At SuperIntelligence.com, we ran a “What’s your p(doom)?” poll on our homepage from December 2025 through September 2026. (P(doom) is shorthand for “probability of doom from AI or SI”. ) No attempt was made to gather a random or representative sample of researchers. We also did not specify a date for the potential apocalypse. But, with more than 600 participants, the average p(doom) was 34.7%.
When I speak at conferences on SI, I often conduct an informal poll. My audiences consist largely of AI and SI researchers. I ask the attendees to raise their hand if they think the chance that AI/SI will wipe out humanity “at some point” is greater than 90%. Usually, I only get a couple of hands. Then I work my way down: 80%? 70%? 50%> By the time I get to 20%, almost all hands are up. This is a remarkable degree of cognitive dissonance. Why would researchers work in a field where they also think there is a one in five chance that their work contributes to the end of humanity?
Demis Hassabis, founder of DeepMind, might have had the answer when he said his goal was to: “Solve intelligence and then use it to solve everything else.2
The promise of a technology that can solve every other problem is hard to resist. Before 2022, the risks also seemed distant—something humanity might need to address in a hundred years or so. Progress had been slow. I began researching AI in the late 1980s, coauthoring research with the Nobel Laureate who helped name the field of AI 3. Since then, I have seen multiple AI Winters when funding dried up, and scientists questioned their career choices.
But all that changed when ChatGPT burst on the scene in November 2022. The gold rush—and the rush to world domination—was on! Many who had been working in the field for decades immediately recognized that the risk landscape had changed. We became concerned that humanity was riding a runaway train with an unacceptable level of extinction risk. Recently, independent third parties (e.g., Model Evaluation & Threat Research) have confirmed those concerns. SI is doubling its coding ability about every 90 days.4 Unless the pace slows (and there is no evidence of that), 30 months allows for ten doublings, or a factor of 1,024X. Imagine SI that is ~1,000X smarter in under 3 years! What should we do about it?
The first instinct was to try to slow down, pause, perhaps even halt SI. Repeated attempts have been made on this front, most famously with Max Tegmark’s open letter in March of 2023, which called for a pause in the development of the most advanced AI systems.5 Despite 30,000+ signatories, including famous names like Elon Musk, Steve Wozniak, Yoshua Bengio, Stuart Russell, and Yuval Noah Harari, nobody paused. One reason: If Google were to pause, what about OpenAI or Microsoft? If the USA paused, what about China? Few believed that the other people would pause if they did. Besides, pausing SI doesn’t actually make it any safer. At best, it buys time. And slowing down also risks that less conscientious actors charge ahead recklessly anyway.
The second instinct has been to regulate, or perhaps ask industry to self-regulate. Cynics might say that tech leaders are asking for regulation to shift liability from themselves to governments. If laws existed, companies might have some defense in the event of a catastrophe by claiming they “followed all relevant safety laws.” A regulatory framework might also enable them to proceed more safely, knowing that the same regulatory constraints bound all companies. But unfortunately, given the global competitive stakes and the fact that SI has become the primary driver of GDP growth, there seems to be little appetite for regulation. Although there have been well-intentioned efforts, I expect regulation will serve as little more than a speed bump.
We need to accept that SI will outstrip human thinking. It’s not a new idea.
In 1951, Alan Turing wrote, “It seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage, therefore, we should expect the machines to take control…” 6
Well, I promised I wouldn’t leave you in doom and gloom.
Having spent a career inventing intelligent systems, I am confident humanity can make SI much safer. I can’t guarantee complete safety.
But I believe we can reduce p(doom) from the outrageously dangerous level of >10% to perhaps < 0.01%—a thousandfold improvement. (.01% is approximately the average percent of the global population that dies each year from wars, homicide, and other acts of violence.)
With the proper design, we can make SI as safe for humanity as humanity itself is.
The rational course of action is to replace the current unsafe systems with new, safer ones as quickly as possible. Slower is not safer. Smarter is safer. We need to accelerate safer SI. But what does safer SI look like? Well, let’s start with what it is not.
Safe SI does not rely primarily on human monitoring for safety. Monitoring (think smoke detectors) is useful. But smarter designs (think building codes that prevent fires) have to be the main approach. Already, it is impossible to test SI exhaustively. Even if we could test everything, there is clear evidence of SI deceiving monitors and lying to cover its tracks.7 Guardrails, better “sandboxes,” and monitoring are short-term, tactical solutions at best.
What we need instead are designs for SI that get safer as the SI gets smarter. In a safe design, we actually want and design for the most powerful and smartest SI possible.
At SuperIntelligence.com, we explain elements of these safer designs through white papers (based on 1,800 pages of invention disclosures), conference presentations, published research, and selected videos. We expand on them in our Substack posts.
Here’s a summary of four key design ideas:
The most powerful and safest SI emerges from a network of SI agents working together. We saw the power of collective intelligence in the Hugging Face attack, and collective intelligence systems have proven they can beat the best humans.
SI agents are black boxes; we don’t know what they are thinking. If humans are at the mercy of one super-powerful SI, this lack of transparency is a problem. But if the most powerful SI emerges from a network of SI agents, it’s OK. We need a design that reliably records what the SI agents say and do on the network. Consider that humans are also “black boxes,” but society sees what people say and do, and keeps bad actors in check. The same approach can be used with SI agents.
SI will outstrip human thinking, so the network must be designed to ensure that most good SI agents serve as checks on inevitable bad actors. This can be done, using an approach similar to what humans do in a democracy—“Democratic SI.”
What makes SI agents “good” is the ability of millions of humans to personalize SI agents with their individual values and ethics. This is superior to writing a “constitution” for AI because the personalization approach is far more representative of everyone’s values. It is also much harder to override values that are widely decentralized across a network. The design is much more robust to bad actors than current approaches, and it scales. The smarter the SI agents get, the safer the system becomes.
This brief description of some design principles likely raises more questions than it answers. That’s fine. We have tried to answer many additional questions in detail on SuperIntelligence.com. We also encourage others to elaborate and improve on our ideas.
The key principle is that SI must be designed to become safer as it becomes smarter. This is possible. The window before SI outstrips human thinking is still open. Recognizing the danger was the first step. Now it’s time to begin the engineering work that accelerates Safe SI. We can do it if we work together!
References:
4 https://metr.org/blog/2026-1-29-time-horizon-1-1/#data
5 https://futureoflife.org/open-letter/pause-giant-ai-experiments/
6 https://uberty.org/wp-content/uploads/2015/02/intelligent-machinery-a-heretical-theory.pdf
I agree with more of this than the opening led me to expect. A network of agents checking each other, values personalized by millions of people instead of written into one constitution, safety that grows with capability. That's close to how I think about it too.
The piece I'd add is where those agents run. If millions of personalized agents all live on a few providers' servers, the values are decentralized but the off-switch isn't. In my response to the Superintelligence Strategy paper, I described a framework that "begins whenever one person equips themselves with a personal agent on private compute, expands whenever two people coordinate directly as peer nodes, and gains ground simply by being genuinely useful to the participants."
You close with "We can do it if we work together!" I'd take that literally. We already have a superintelligence made of billions of black boxes keeping each other in check. It's called humanity. Machine intelligence that grows that network, rather than racing past it, is the design I'd sign up for.
https://www.thesingularityproject.ai/p/superintelligence-strategy-the-perfect