Zuckerberg Brings the World Closer to Democratic SuperIntelligence
Mark Zuckerberg has moved remarkably close to a democratic model of SuperIntelligence.
This week, Mark Zuckerberg published The Future Is for Everyone, the clearest statement I have seen from the CEO of a major technology company about how SuperIntelligence should be built.
I found myself agreeing with a surprising amount of it.
Zuckerberg rejects the idea that humanity should place its future in the hands of one centralized, supposedly benevolent SuperIntelligence. Instead, he argues for a personal SuperIntelligence distributed broadly among billions of people, with different agents representing each person’s goals and values. Those competing interests would check and balance one another, much as they do in democratic societies and markets.
I was very happy to see a company with the resources and reach to bring personal SuperIntelligence to billions of people embrace this path. It feels like a watershed moment!
Zuckerberg gets three big things right
First, there is no singular set of human values that one SuperIntelligence can represent.
Humanity is not a monoculture. Humans disagree about politics, religion, morality, priorities, risk, fairness, and even what constitutes a good life. A single SuperIntelligence aligned with one organization embodies someone’s values, whether those of the developers, the government, or the people who wrote its safety rules. If produced by any one company, the values are unlikely to be broadly representative of the values of all of humanity.
Second, alignment should be with individuals’ values rather than with the institution that built the AI.
This may be the part of his essay I found most encouraging. Today’s dominant alignment approaches ask a relatively small group of people to determine how an AI should behave. Even Anthropic’s well-intentioned “Constitutional AI” requires a constitution written by a relatively small group of humans working at a single institution.
Ethics cannot be determined logically. Unlike facts, they have to come from a subjective source—typically people, families, communities, and cultures. My approach has been to design SuperIntelligent systems in which millions of personalized AI agents learn from millions of different humans, so that each perspective contributes to a much broader representation of human values than would be possible if one company, government, or committee were to define morality for everyone.
Zuckerberg similarly argues that a personal agent should serve the user’s goals and values rather than the divergent goals of the company that created it. On this principle, we strongly agree.
Third, SuperIntelligence safety is partly a balance-of-power problem.
Zuckerberg argues that the significant majority of intelligence should remain directed toward people’s goals, and that multiple SuperIntelligences should be able to check one another. This is similar to some of our designs, which use blockchain-inspired methods to ensure integrity. A decentralized system becomes much harder for a single bad actor to dominate when the majority of the network’s power remains outside that actor’s control. Similarly, a SuperIntelligent system composed of millions of individually personalized Superintelligences can rely on the majority of the actors in the system to keep the relatively few bad actors in check.
A community of SuperIntelligences can provide checks and balances that a single SuperIntelligence cannot. This feature becomes increasingly important as the speed of non-human thought increases. If an AI, or SuperIntelligent, system can reason through what amounts to years of human thought in moments, then waiting for humans to notice dangerous behavior is not an adequate safety architecture. Instead, safety has to scale with the speed of the SuperIntelligent system itself. That probably means that the many “good” SuperIntelligences will have to keep the few “bad” ones in check. Humans will not be able to think fast enough to do the job on their own.
Safety by Design
Although Zuckerberg did a good job of outlining some of the big picture considerations, when it comes to safety, the devil is in the details of the design. As mentioned, we want a design in which the safety is considered and checked, no matter how fast the system thinks. Fortunately, some of AI’s pioneers provided a detailed architecture that might fit the bill.
Herbert A. Simon and Allen Newell (two of the scientists who helped name the field of AI way back in 1956) described how problem solving (or, more generally, “thinking”) can be represented as a structured search through a problem space. All problems have a goal. Solving a problem usually requires setting additional subgoals, determining possible actions, making decisions, gathering new information, moving from one state to another, and thinking progressively. So any serial thinking processes can be modeled as having steps, typically including setting goals and taking actions.
In the variant of Simon and Newell’s architecture described in my white papers, ethical evaluation can be incorporated into the thinking process itself.
For example, every time an AI agent, or Personalized SuperIntelligence, proposes a new goal or subgoal, that goal can be evaluated.
If the AI’s sequence of steps in its thinking process begins producing suspicious patterns, those patterns can be monitored and evaluated.
One mildly concerning action might trigger a warning; multiple warnings might prompt greater scrutiny, and a sufficiently serious series of potentially dangerous steps could halt the thinking process or trigger an audit by humans or other specialized AIs.
This sort of design matters because dangerous intent does not always appear in a single obvious request. Someone planning harm may break the objective into ten individually innocent steps. Any one step tells you very little. The sequence tells you much more. External monitoring systems can maintain context and look for patterns. A mechanism embedded in the thinking process has direct access to the evolving goal sequence while thinking is still ongoing, before consequential actions are taken.
Because the checks can be embedded in the thinking process, they can scale with the system rather than depending on human review operating outside it. That is an example of what I mean when I say safety must be designed in rather than tested in afterward. We cannot assume humans will remain fast enough to supervise a SuperIntelligence from the outside.
A Few Areas Where Zuckerberg and I Still Disagree
First, human capabilities will not increase quickly enough to keep pace with SuperIntelligence simply because everyone has access to a personal agent. We have to design a network in which millions of personal SuperIntelligences can interact safely. This network must allow the SuperIntelligences to check each other and flag potential issues for human review. The system ideally should also incorporate democratic principles by design.
Second, cybersecurity and biological risks may require more serious design protections than Zuckerberg proposes. When the main line of defense is limiting access to hazardous chemicals, the assumption appears to be that there is no way to design AI to have good intentions, and so we have to rely on restricting access – a poor substitute. We can do better.
Third, while competition among Meta, Google, OpenAI, Anthropic, the United States, and China obviously matters today, the larger long-term question is not which company or country wins. Rather, it is whether humanity collectively maintains control over intelligence far greater than our own. Zuckerberg acknowledges the issue but does little to address it. Since the danger of SuperIntelligence to all humans likely dwarfs the concern that one company or country has a momentary advantage over another, we cannot afford to sidestep the most important safety concern facing humanity.
On the central point, however, we agree: a single centralized SuperIntelligence is not the safe answer. That alone represents a major shift in the public conversation.
Why META Might Be Best Positioned to Achieve Personalized SuperIntelligence
Personalizing an agent to a specific human requires knowing a great deal about that human. Meta already has unusually rich, long-term signals about people’s interests, relationships, and preferences across its platforms. In several of my own patents, I described a one-click personalization method in which a user presses a single button and receives an agent that already reflects their interests, goals, and preferences. I can only assume that META is working on, or has already developed, similar approaches. With the possible exception of Google, no other company is better positioned than META to create a “one click” personalization for SuperIntelligence. While Anthropic and OpenAI struggle to learn about users by gathering interaction data as quickly as possible, Meta and Google (and perhaps, to a lesser extent, Microsoft via LinkedIn) already have a huge head start because of their vast amounts of user data.
All of this makes Meta’s architectural choices unusually consequential. If personal SuperIntelligence reaches billions of people, the key question becomes: what safety mechanisms are built into the agents and into the network on which they operate?
What Meta should build next
Zuckerberg says he does not believe any one person, including himself, should determine how SuperIntelligence is deployed. I agree. Meta is proposing governance and independent oversight of model releases. This may be useful, but simply reviewing a model before release leaves more fundamental architectural questions unanswered.
The challenge for Meta Superintelligence Labs, and every other lab developing increasingly autonomous agents, is to design a system that can scale safely. Some of the key design questions we mentioned include:
Where in a SuperIntelligence’s thinking process are its goals evaluated for ethical intent?
Whose values inform that judgment?
Can the system recognize a harmful pattern spread across many individually harmless steps?
What automatically triggers greater scrutiny or human intervention?
Are the safeguards designed to scale as SuperIntelligence thinks much faster than humans trying to oversee it?
These, and other, design questions need to be addressed as quickly as we can.
An open invitation
My SuperIntelligence design papers are publicly available at superintelligence.com.
They describe approaches to distributed intelligence, personalized agents, democratic representation of human values, ethical evaluation in problem-solving, human oversight, and other mechanisms intended to reduce the risks posed by SuperIntelligence.
They are freely available for responsible research and implementation.
If Meta Superintelligence Labs finds something useful in them, I hope they use it. If another lab finds something useful, they are free to use it as well. If researchers find something wrong, I hope they challenge the designs, improve them, and make the improvements available to everyone. The stakes are far too large for this to become a contest over who gets credit. Similarly, the business opportunities are so large that multiple companies and countries can win. In fact, the only way humanity loses is if collectively we are too shortsighted to recognize that we need to work together to ensure safer designs for SuperIntelligence.
I have spent years arguing that humanity should not bet its future on a single opaque, centralized SuperIntelligence controlled by a single organization.
Now the CEO of one of the world’s largest technology companies is publicly making a similar case himself.
I am heartened.
The probability of an excellent outcome for all humankind has just increased greatly.
Bravo!
Dr. Craig A. Kaplan is CEO of iQ Company and Founder of Superintelligence.com, where he designs safe and ethical AGI and SuperIntelligence systems. He earned his PhD at Carnegie Mellon, co-authoring research with Nobel Laureate Herbert A. Simon, and previously founded PredictWallStreet. He holds numerous AI-related patents.