When AGI Becomes SuperIntelligence, Testing Is Not Enough
A Personalized SuperIntelligence can become more intelligent than the humans who designed its safeguards. Testing alone cannot guarantee its safety.
Safe SuperIntelligence by design is the essence of the approach described here. Testing works when the people running the test can anticipate what the system might do, and that condition holds for the AI we have now. Artificial General Intelligence, or AGI, is AI that can do any intellectual task as well as, or better than, the average human. A Personalized SuperIntelligence, or PSI, does not stop there.
Because PSI is self-improving, it will become exponentially more intelligent than its creator. Any initial checks and safeguards designed by humans will quickly become ineffective in the face of a PSI’s superior intelligence. Rather than relying on testing to ensure safety, we envision systems in which PSI safety is built in.
Think of your PSI as your personal, super-smart AI agent. It does your bidding and learns about you and what you want over time. It can be far more effective and efficient than you are at most everyday online tasks. It offers excellent advice. And best of all, it knows you, can relate to you, and thinks like you.
Soon, general SuperIntelligent AI will be widespread. On Meta’s earnings call in July 2026, Mark Zuckerberg described the company’s strategy as putting superintelligence directly into people’s hands, and Meta stated it is progressing toward bringing personal superintelligence to everyone.
Many other companies are also working feverishly in a race to be the first to release powerful and easy-to-use systems for everyone. I suspect that within the next six to 18 months, extremely easy-to-use systems, for example, pressing a single button to personalize your PSI, will be released by Meta or any number of companies. I believe that eventually, almost every individual human on Earth will have a PSI that acts on their behalf.
What Meta has announced is a product. Nothing in these announcements addresses how a system that outgrows its owner stays safe.
The way a PSI grows is worth understanding because it explains both its benefits and its dangers. If you want to expand the capabilities of your PSI, it is as simple as trading data with a friend or buying or selling data on an online marketplace, then integrating that data into your PSI to provide additional intelligence, skills, and knowledge.
Over time, your PSI can acquire the wisdom of billions of humans and AI agents. After acquiring that knowledge, your PSI can run simulations based on your goals and purposes to improve further and act on your behalf, with your permission, as you see fit.
Consider what that makes possible for one owner. Your material wealth can be multiplied by your PSI, acting as your investment advisor and agent. Your PSI can be cloned and leased out. A thousand versions of it can handle a thousand different errands and missions for you simultaneously, under the direction of other cloned PSIs that are also under your direction and serving your interests!
For society at large, multiple PSIs, pooling their intelligence and participating in a PSI network, can serve as a Planetary Intelligence, providing benefits for all people and our planet.
Of course, the properties that make a PSI valuable are the same ones that make it dangerous. Because of their extreme intelligence, PSIs also represent a dangerous potential threat to human safety. Its superior intelligence will be highly effective and efficient at achieving goals that reflect your values, whatever those values are. That is why the source of a PSI's goals and values is a central safety question.
A PSI may begin with values already present in its base model, but its customization depends on its owner. Those values help determine the goals it pursues and how it pursues them. The next post takes up how a PSI learns its owner’s values, both from what the owner teaches it directly and from watching what the owner does, and why a community of other PSIs is still needed to keep any single SuperIntelligence in check.
This series draws on White Paper 5: Safe Personalized SuperIntelligence. Read it in full to see how every piece fits together!
If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.