Why Safe AGI Needs Many Humans, Not Few
The inconsistency of human values is the strength, not the weakness.
Human values are inconsistent and contradictory. That is why an AGI built on the values of millions of humans is safer than an AGI built on the narrow values of a small group. The inconsistency is the strength, not the weakness.
A clean set of rules has one author or a small group of authors.
Whoever they are, however well-intentioned, they cannot represent the moral experience of 8.3 billion humans.
Their blind spots become the system’s blind spots.
Their biases become the system’s biases.
If the rules are flawed, bypassed, or overruled, the entire system is compromised because the values live at a single point of failure.
A value system drawn from millions of diverse humans has no such single point. It is more representative, more resilient, and more resistant to manipulation because it carries the contradictions of human moral experience inside it.
This is the second core idea of White Paper 2, after “heart before head.”
Values must come from humans. The question is how many humans, with what diversity, are contributing in what way.
Most people imagine AGI as a single, very powerful AI system that crosses some threshold of capability and becomes generally intelligent on its own. A single entity, a single mind, with a single set of values, chosen by whoever designed the system.
White Paper 2, Ethical and Safe AGI, proposes something different. AGI does not have to emerge from a single system. It can emerge from the collective intelligence of many intelligent agents, both humans and AIs, working on a shared problem-solving network. No single AI agent on the network needs to possess general intelligence. General capability emerges at the level above any one agent, through the work of many specialized contributors, human and AI, coordinating across domains.
This is how human civilization already works. No single human can design a complex microprocessor, perform heart surgery, write classical music, adjudicate complex legal disputes, and navigate a container ship across the Pacific. We function as a civilization because each of us does a small piece of the work, and the rest of us trust the people who do the other pieces.
Collective intelligence works the same way. Each participant contributes something unique, and those contributions compound. One participant may be a travel agent with deep knowledge of international logistics. Another may be a plumber with decades of practical expertise. A third may be a physician, a fourth a lawyer, a fifth a software engineer. Each customizes their own AI agent, an AAAI or Advanced Autonomous Artificial Intelligence, with their specific knowledge, skills, personality, and values. Individually, each contribution is modest. A single user’s tweak to a base AI model may not, by itself, produce a dramatic improvement. Collectively, the contributions of millions of users can take a base model to AGI-level intelligence.
I have built systems that work this way. In the 1990s, I founded a company that combined the intelligence of millions of retail investors. By 2018, PredictWallStreet was powering one of the top-ten performing market-neutral hedge funds in the world. Millions of everyday investors, working collectively, outperformed Wall Street’s top traders. Not because any one of them was smarter than a hedge fund manager (most were not), but because the collective signal was more accurate than any individual signal could be. The same principle works for AGI. Many everyday contributors, integrated well, can outperform a small group of experts.
The same logic applies to values.
Most ethics writing treats the inconsistency of human values as a problem to be solved. Different cultures hold different views. Different religious traditions disagree. Different individuals weigh competing claims differently. The standard response is to look for a single coherent framework that can resolve the disagreements. The collective intelligence approach inverts that. The diversity is the point.
There is already broad agreement on certain principles. Most humans agree that human life is precious, that unprovoked violence is wrong, that honesty is generally better than deception, and that the suffering of innocent people should be minimized. What makes ethics complex is everything that lies between these general principles and the special circumstances in which they are tested. If we want AGI to have values aligned with human values, we must provide as large and as diverse a sample of human ethical reasoning as possible, continuously updated as new situations arise.
A small group cannot do this. Whatever the small group decides reflects that group’s blind spots, biases, and historical moment. Anyone disagreeing with the small group has no path into the system. A representative system has many paths in, by design.
The next post takes up the architecture that makes this possible: the five subsystems of SCAN-II.
This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!
If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.