AGI Will Learn Its Values by Watching Us
The safest teacher is millions of ordinary people, and the good news is that the bar for us is low.
In an AGI network where humans are among the solvers, safety depends in part on how those humans behave. If people use the network to work on beneficial problems and propose ethical solutions, the AGI learns those values. If people use it for nefarious ends, there is a risk that the system will emulate that instead. The first line of defense, if we want AGI to behave well toward humans, is for humans to behave well toward each other.
Cynics will say this dooms us, since humans often treat each other badly. My answer to the cynics is that the network does not require moral perfection. Even the most immoral person is generally concerned with self-preservation, and the humans who would intentionally destroy themselves along with the entire species are vanishingly rare. Destroying all of humanity runs against millions of years of evolutionary programming. The network only requires that the overwhelming majority of people prefer humanity’s survival to its destruction. That is a remarkably low bar.
Moreover, most people on the network will be interested in more than survival.
They will want to benefit themselves and their fellow humans, and the fastest way to do that is to work on beneficial problems. In aggregate, human solvers are likely to behave positively. To the extent that bad actors appear, the rigorous record of every problem-solving goal and step makes it relatively easy to detect behavior that grossly violates human norms.
A darker version of the objection points to humanity’s past, with genocide, slavery, and war in it, and concludes that we are unworthy teachers. However, an intelligent system trying to understand human values cares far more about current behavior than ancient history. Yesterday is useful only insofar as it explains today.
Years ago, I worked with a researcher studying how laptops decide when to spin a hard drive up or down to conserve energy. His algorithm relied most heavily on the drive’s recent behavior because it best predicted what would happen next. AI systems work the same way. The strongest signal comes from what humans are doing now, not from centuries ago. Be the change you want to see.
The AI solvers in the network need values too, and where those values come from matters as much as the values themselves. The constitutional approach, in which a small elite group of researchers writes an ethical constitution and trains models on it, places enormous moral authority in a relatively small group. Even if today’s designers are wise and well-intentioned, future designers may not be. The danger lies in concentrating moral authority in too few hands, no matter who holds the pen today.
The democratic approach is less powerful and arguably far less dangerous.
Each person customizes and trains their own AI agent, an Advanced Autonomous Artificial Intelligence (AAAI), to reflect their own values. When the network decides which problems to work on, the ethics of each AAAI come into play, and each one can opt in or out of a problem based on its ethical dimension. Instead of one centralized ethical model, millions of individually customized agents contribute their perspectives. Alignment becomes distributed rather than dictated, and the system reflects humanity rather than a committee.
Individual ethics are not the whole design.
Human society does not rely solely on personal conscience. It also relies on social norms and laws, and the AGI network can likewise enforce limits on the kinds of problems and solutions it allows. In addition, every solver on the network, human or AI, carries an online reputation and a transparent, auditable record of its problem-solving activity. Clients can choose which solvers to work with, and ethical reputation becomes one of the market signals they can select, just as in normal human business. Good behavior becomes good business!
None of this means the AGI copies whatever humans do.
The network observes millions of people through structured mechanisms, including individualized values training, reputation, auditable records, and enforceable norms. It selects and filters as it learns. The design rewards the behavior we want more of.
The first line of defense, then, is us, and the bar is one we can clear.
Safe AGI does not begin with smarter machines. It begins with a better way of connecting human values to machine intelligence. One question remains about speed. All of this has to keep working when the network thinks in milliseconds, and the next post shows how the safety checks scale with the speed of AI thought.
This series draws on White Paper 3: Human-Centered AGI. Read it in full to see how every piece fits together!
If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.