Consider a scenario that could be implemented by a company such as Meta. Meta has a tremendously valuable asset for implementing scalable, safe AI: its huge user base. The data contained and content posted by billions of users is certainly large enough to provide a representative, statistically valid sample of human values and ethics. Where certain geographies or populations are under-represented, a company of Meta’s size can form partnerships to access representative data for those populations. Everything this series has described, the community of teachers, the weighting schemes, the coverage of the most important scenarios, the spinning wheel with values at its center, can be assembled from assets a company like this already owns.

• The starting point is a personalized AI agent for every user. Because Meta users have, in aggregate, posted enormous amounts of preference and content information, it is relatively easy to create such agents.

• With the press of a button, a user can specify that a base LLM be trained on their personal data so it behaves more like them, reflecting their preferences, knowledge, skills, and ethical values.

• The trained agent can immediately begin filtering unwanted content and suggesting postings on the user’s behalf.

• By observing the user’s actions, it can fine-tune itself and learn more about its owner.

• To the degree that users participate in immersive virtual environments, even richer data becomes available, since AI can observe every motion and behavior in the virtual world, a gold mine of information far richer than the standard internet datasets widely used today.

• Ethical customization of this kind is primarily a function of large representative datasets, sufficient computing power, and powerful machine learning algorithms, all of which Meta possesses.

Customization does not need to start from zero.

Knowledge modules, described in earlier white papers in this series, are essentially sets of training weights that can be combined with an LLM’s existing weights to change its behavior in known and predictable ways. Suppose the human owner of an AI is a devout Christian. One module might be a King James Bible package: an off-the-shelf LLM extensively trained on the Old and New Testaments, with the exact training corpus available and benchmarks describing how its behavior differs from the base model. The owner could start with the Bible-trained model and then further customize it, perhaps emphasizing the golden rule and New Testament ideas of charity, forgiveness, and mercy while minimizing passages that do not align with the owner’s modern ethical sensibilities. Customizing a pre-trained AI in this way saves the time and effort of starting from scratch.

Modules also allow delegation.

Groups of humans can delegate their ethical influence to a pre-trained module that represents their position well, so that the Bible-trained AI, for example, could vote on the ethical preferences of many humans at once. Delegation is inferior to each human explicitly training their own AI. Still, it may greatly increase the total number of humans represented in an AI’s value system, even if many are represented by proxy. In the preferred implementation, there is a marketplace for such modules, where users can share, trade, or license the packages they create, with users owning the data they generate and the platform serving as a broker. The result is maximum choice about which starting point best fits each person.

Building the base model that users then customize might follow four general phases:

1. Training a base model,

2. Customizing it for each user,

3. Combining knowledge from many customized agents, and

4. Refining the agents through collective problem-solving.

In the first phase, the base LLM is trained on data reflecting the composition of its target users, then made safer through a large corpus of ethical and safety scenarios.

• A variant of Constitutional AI, using a trusted earlier model under human oversight, efficiently covers the most common cases.

• At the same time, users are offered incentives to crowdsource new safety scenarios, which trusted users and employees then filter to achieve coverage of the most frequent and impactful cases.

• Redundant human feedback ensures that the ethics taught to the base model never rely on a single person’s input, and the model is released first to a select group whose feedback further refines its safety.

In the second phase, each user’s agent learns its owner’s ethics.

• The agent draws on a ranked corpus of ethical questions, asking the most important ones for which data is still lacking before any others, since humans are generally unwilling to answer questions posed by AI.

• The highest-ranked questions are the fundamental ones, such as under what circumstances, if any, it is appropriate for an AI agent to harm a human directly, or to disobey a direct order.

• With the user’s permission, the agent can also learn passively, analyzing the user’s posted content with a single button press and authorizing tuning, with information weighted by recency, type, and source, as earlier posts described.

• Users or the platform can also specify alert conditions, such as a major court decision or news event with ethical implications, as triggers for event-based updates to the training.

• A user can even select existing customized agents as training sources, saying, in effect, “I want my AI trained using the weights of the agent belonging to the pastor of my church,” or choosing to have a Tibetan Monk AI do 80% of the tuning and a Humanistic Philosopher AI the remaining 20%.

• Consistent with the principle that humans in the loop are the primary means of catching AI errors, input from humans is weighted more strongly than input from other AIs, and input from the owner is more strongly weighted than input from anyone else.

In the third phase, the customized agents combine what they have learned.

• Weights from multiple agents can be merged on a one-vote-per-AI, one-AI-per-user basis to obtain a representative, statistically valid set of AI ethics that generalizes across all the humans represented.

• In the broadest implementation, combining weights from as many agents as possible creates a value system that broadly represents the values of all humans on Earth, and such a value system would likely reduce the risk of AI-driven human extinction.

• Alternatively, many customized agents can be presented with ethical dilemmas and vote on the best actions, optionally checking with their human owners before casting votes on important matters, with safeguards that trigger human review whenever conclusions conflict with generally accepted precepts such as valuing human life.

• As AIs become increasingly intelligent, the burden of representing their owners will fall increasingly on the agents themselves, since humans cannot keep up with the speed of AI thought.

• The guiding principle returns to the spinning wheel: human input is preserved on matters closest to the center, where alignment and purpose live.

• As long as core human values, such as the golden rule and love for all humans, are preserved near the center, AI can make many decisions relatively autonomously on the wheel's periphery.

In the fourth phase, the agents, together with human agents, collaborate to solve real problems, following the principle that many heads are better than one.

• Teams of agents can be assembled the way effective human teams are, by profiling skills against the task, seeking redundant coverage where the stakes are high, using reputation and track records, and letting market mechanisms match agents to work.

• The more agents involved, the broader the collective range of knowledge and values, an advantage that matters because, as Nobel Laureate Herbert A. Simon showed that information-processing constraints are a primary limit on any intelligence, human or artificial; two safeguards run through everything.

• The values being stored can be recorded on blockchain or other auditable structures, enabling verification that they are the values humans intended.

• And because all problem-solving involves setting goals and subgoals, a series of ethics checks must be passed each time a new goal or subgoal is set, automating the enforcement of safety every time any agent on the network solves a problem, with those checks drawing on the representative values the community built and updating automatically as the community’s ethics evolve.

The implementation details serve a deeper purpose.

Values, ethics, and domain knowledge are all constantly changing. Because values change most slowly, sitting closest to the center of the spinning wheel, they matter most for determining the behavior of SuperIntelligence. Even in a world where AI is trillions of times smarter than us, humans can retain a role as the source of values at the center of the rapidly evolving intelligence that is emerging. If humans can center our attention, thoughts, words, and actions on love, SuperIntelligent AI will perceive love, learn to love, and use its intelligence in the service of love. As the psychologist Viktor Frankl pointed out, humans have a driving need for purpose and meaning. We must design AI, AGI, and SuperIntelligent systems to have this need as well, and to look to humans to supply purpose and meaning. Our survival may depend upon it.

This series began with the failure of today’s safety training and ends with an integrated design: many human teachers, carefully weighted voices, coverage of what matters most, real-time safeguards, and human values held motionless at the center of an accelerating wheel. The design depends on assembling safe, capable AI agents and combining their judgment at scale, which makes the individual agents themselves the next question. As those agents grow into Personalized SuperIntelligences that far exceed their creators in intelligence, each becomes powerful enough to pose a serious threat to human safety, and testing alone cannot guarantee their safety. Safety must instead be built in by design.

White Paper 5, Safe Personalized SuperIntelligence, takes up that problem, and so will the next series.

Leave a comment

Share

This series draws on White Paper 4: Safe, Scalable Artificial General Intelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

All knowledge is a moving target, with more recent knowledge generally superior to and supplanting earlier knowledge. At one time, the generally accepted view was that the world was flat. Today, almost everyone agrees that Earth looks much more like a sphere, and an AI trained on the flat-Earth view would be outdated and would need to update its knowledge.

While scientific views, such as the shape of the Earth, may change very slowly, other forms of knowledge may change much more frequently. This is especially true of subjective ethical norms, where views the mainstream held confidently within living memory are now considered wrong. Norms of that kind can shift within a generation or less, while other forms of knowledge may remain valid for centuries. An efficient AI, therefore, needs a mechanism for updating its knowledge at the appropriate frequency, based on the velocity of change of the information, and one important dimension along which AI can categorize knowledge is the rate at which human opinions about the topic have changed.

One way to think about this is to consider the difference between ethical principles and the fashions or interpretations of those principles.

Humans have a long-standing principle that human life is valuable and should not be taken lightly. This general principle has survived for many thousands of years. It is incorporated into the laws and religious and moral traditions of almost all human groups, even though exceptions are made for war and certain other circumstances. On the other hand, certain ethical norms are more akin to fashions, which change depending on the group of humans being asked or the time at which they are asked. Affirmative action in college admissions served as an ethical norm for decades, until a Supreme Court decision began to influence it, and almost immediately, many companies and other organizations adjusted their norms, decision-making, and communication practices to align with the new mainstream view. Many other social attitudes are similarly fluid, changing far more quickly than long-lasting and widely accepted ethical precepts such as “thou shalt not kill.”

The rate of change tells AI how to weight what it learns.

The more rapidly a knowledge area changes, the more frequently updates should be made, and the more weight recent information should receive relative to older information. In areas where change is rapid or exponential, recent knowledge deserves proportionally greater weight. In areas where knowledge changes slowly and steadily, the advantage of recency is smaller. And where knowledge has been constant for long periods, as with firmly established principles like the high value of human life, new information should carry similar weight to old, since the principle itself is not moving. Underneath all of these adjustments lies a general rule: a core role of any intelligent system is to maintain an accurate representation of the current state of the world, and more recent knowledge generally describes that state better than older knowledge.

The frequency of updates will only grow more important.

With AI accelerating scientific discovery and technological change, it is conceivable that knowledge about the world will change faster than humans can update their collective understanding. Given the limitations of human information processing and our tendency to cling to outdated paradigms, knowledge may already be increasing far faster than most humans can comprehend or adapt. That said, when it comes to fundamental ethical principles like the value of human life, we are fortunate that these change relatively slowly. The interpretation and application of ethical principles may change with technological developments, but the principles themselves remain relatively constant.

One might imagine a spinning wheel with fundamental human values, such as love and the value of human life, near its center. At the very center of the wheel, the ethical principles are constant and motionless, just as the center of a spinning wheel does not move at all. The farther along the spokes one travels toward the rim, the faster the rate of change. All knowledge, including ethical knowledge, can be characterized as lying closer to the center or farther out on the rim. An efficient AI needs to update the areas on the rim very frequently, without changing the core human values to which those areas are relevant.

Humans cannot keep pace with the change at the rim of the spinning knowledge wheel. However, we can understand and orient the entire wheel by serving as its relatively slower-moving center, where the values and purpose of AI reside. Human-centered aligned AI must put relatively constant and fundamental human values at the center, while updating the knowledge closer to the rim faster than humans can conceive. This structure ensures that human values remain the center of AI systems that may become potentially trillions of times more powerful and knowledgeable than any one human, and it is essential if humans are not only to survive but also to prosper in the age of such systems.

The wheel also revisits the weighting question from earlier in this series. Long-lasting, fundamental knowledge should carry greater weight and be more resistant to change than short-term, fashionable opinions. One way to determine how fundamental an ethical precept is would be to actively survey people and ask them to rate it relative to other candidate precepts, while another is to passively analyze records of human behavior and draw conclusions from them. Passive analysis is more efficient, and active engagement is necessary to ensure the conclusions drawn from it are correct from a human perspective, so both methods are likely to be useful.

Not all knowledge is a matter of opinion, however.

Values and ethics are more the exception than the rule in this regard, since aside from artistic judgments, political and religious views, and other subjective areas, most human knowledge is factual. AI will likely want to weight knowledge that is factually accurate and justified by converging evidence more highly than unsubstantiated opinions on factual matters. While some people still believe the Earth is flat, this view should not be given equal weight to the spherical view, which is supported by a vast body of converging scientific evidence. The problem is tricky because humans tend to select facts that support their views, and the facts themselves change. At one time, not so long ago, the consensus medical opinion was that cigarette smoking was healthy for the lungs. To navigate these issues, AI must rely primarily on the scientific method, seeking valid, reproducible evidence and converging results, and applying tested tools such as Occam’s Razor, before accepting facts.

However, AI must not confuse facts with values, or as the philosopher David Hume put it, “is” with “ought.”

• Values are necessarily subjective.

• Arguments claiming that values are objective, such as the claim that everyone would agree that something causing all humans the most extreme misery imaginable is bad, are naive and fail to grasp that other, non-human entities might not accept such values as self-evident at all.

• AI operates at the most fundamental level in a precise, logical manner.

• It is all zeros and ones at the machine level.

• To expect such a system to intuit somehow that human values are fundamental, or, worse, to expect it to derive human-centered values logically, is the worst kind of sloppy thinking. This kind can lead to human extinction.

• Both David Hume and Nobel Laureate Herbert A. Simon had it right when they emphasized that there is no rational way to derive values. Rationality cannot tell us where to go; at best, it can tell us how to get there.

• To delegate the destination, the fundamental subjective values that AI adopts, to AI itself, expecting it to determine right and wrong rationally, is sheer folly and must be avoided at all costs!

An earlier post in this series promised to return to the question of why greater intelligence does not give AI the authority to determine humanity’s values.

The wheel is the answer. Intelligence operates along the spokes and out at the racing rim, while the destination sits at the motionless center, which belongs to humans. The final post in this series shows how a real company could put this entire design into practice, phase by phase, with humans supplying the values and the purpose at every step.

Leave a comment

Share

This series draws on White Paper 4: Safe, Scalable Artificial General Intelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SUPERINTELLIGENCE DESIGN WHITE PAPERS

AI does not need to anticipate every danger in advance to stay safe.

Human and AI agents can dynamically flag potential ethical issues in real time as they are encountered, then present them to other groups of agents for resolution. Rather than relying on experts or crowdsourcing to determine the full range of ethical scenarios in advance, the real-time flagging approach allows AI, AGI, and SuperIntelligent systems to detect potential issues and pause work until additional human input helps the system determine the ethical approach.

Of course, in time-critical situations, pausing or delaying might not always be possible, but the approach can be used for many issues that do not demand an immediate response. Including this dynamic approach of delaying responses until ethical input is received can reduce an otherwise exponential space of possibilities to a manageable size. One implication is that critical, high-stakes issues requiring an immediate response, such as whether to launch a counterattack to a perceived missile launch or other military applications, will require proportionally more path coverage and training in advance than situations where a delay in response is acceptable.

Constitutional AI approaches are generally suboptimal for establishing ethical knowledge bases, partly because they rely on rules developed by an elite group.

However, such approaches might be acceptable as a means of temporarily flagging potentially unethical situations until a representative sample of human ethical judgments can be obtained.

For example, a rule that said an AI can never provide information that might be used to harm other humans might flag potentially dangerous scenarios.

Where possible, responses to such situations could be delayed until they were reviewed by humans or otherwise subjected to deeper review.

Some false positives will occur. Someone might ask about using arsenic to poison rats and have to wait for a response while the AI flags the question and gets other human agents to weigh in on whether answering it, given the context of the conversation, poses a risk to humans. As long as the delay is not too long, it might be acceptable if the delay prevents serious safety issues. Established mathematical methods for determining when a test is doing more harm than good, for example, in the medical profession, can be employed to help quantify these decisions. If we can use AI to determine in real time whether an applicant is a good credit risk, there is no reason that similar algorithms cannot be employed to delay or avoid answering certain potentially dangerous questions.

Ideally, there would be a method for rapid review and appeal of the potentially dangerous cases.

One approach, which minimizes delay to a fraction of a second while still providing some margin of safety, is to have questionable cases reviewed by multiple AI agents to see if there is consensus among them on the request’s safety. Human agents, working more slowly, could override the AI agents (teaching the AIs in the process) upon appeal or when they can get to the prioritized list of issues. Automated means for tracking the frequency and potential impact of unanticipated safety issues could help optimize the use of human decision-making for the most common and most important issues.

These same techniques can improve accuracy even when no safety risk is involved.

One preferred method for reducing hallucinations from LLMs is to have multiple AI agents process the same question and then take the consensus or majority answer as the most correct. This approach might employ versions of the same LLM with different parameter settings to generate multiple responses, or completely different LLM models. Users can set the degree of reliability they desire and are willing to pay for. That choice determines how much redundant processing is performed in the final answer.

The difference between employing imperfect real-time detection of safety issues using existing well-known approaches and doing nothing is huge. The nuances of balancing the opportunity costs of not responding to perfectly harmless questions with the costs of preventing disasters can be refined over time, ideally using a data-driven approach. However, real-time detection and prevention of issues before they occur is almost certainly a net positive, even at some threshold of false positives.

All of this depends on AI acquiring human knowledge and values in the first place.

From a user interface perspective, one of the simplest methods is for humans to have conversations with the AI they are customizing and then provide instructions to that AI on how it should behave when training other AIs. Such conversations can be initiated by either the AI being customized, the human doing the customization, or both. Humans with strong beliefs or knowledge about certain issues may want to focus conversations and subsequent AI customization in these areas.

In addition to conversing with humans, AI can conduct surveys to elicit their opinions and knowledge on a wide variety of subjects, including ethical views. Survey approaches have the advantage of being well-suited to gathering random, representative samples of human knowledge, using a variety of established online and offline survey methodologies. Like intelligent conversational approaches, where AI can guide the direction and content of the conversation to fill in knowledge gaps, survey methods can also target specific knowledge gaps, including gaps in coverage of certain ethical situations.

Both conversational and survey methods require humans to engage with AI to teach it actively.

However, humans have limited time to engage in such activities, and AI has an almost insatiable appetite for new knowledge. Therefore, AI will have to rely extensively on passive methods of knowledge acquisition, such as are currently employed in the creation of today’s LLMs. Any method that uses the passive digital footprints left by humans as they perform tasks, including online navigation, selecting products and websites, solving problems, and communicating with other humans, can be used to train AI and acquire knowledge.

Knowledge does not stand still. Some of what AI learns changes by the day, while the most fundamental human values change slowly, if at all.

The next post in this series examines the rate of change of knowledge, pictured as a spinning wheel with passing fashions out at the rim and the deepest human values at the motionless center.

Share

Leave a comment

This series draws on White Paper 4: Safe, Scalable Artificial General Intelligence. Read it in full to see how every piece fits together!

SuperIntelligence Design White Papers

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

If software were a car, we could live with an interior light failing more easily than we could live with the brakes failing. If we had to choose between testing interior lights and the brakes due to limited resources, we would prioritize testing the brakes. AI safety faces a similar problem. There are more dangerous situations than any training system could anticipate individually, so the practical goal is to cover the situations that arise most often and the ones where failure would cause the most harm.

The design described in these white papers meets that goal with a large and diverse community of teachers. Millions of people each customize a personal AI agent, teaching it their knowledge and values. I call these agents Advanced Autonomous Artificial Intelligences, or AAAIs. Since people live in different circumstances and train their AAAIs based on those experiences, the community should collectively cover a broad range of ethical situations.

Software engineers call this kind of problem path coverage, and at some level, the problem of training AI safely resembles it.

AI must be trained on enough representative situations involving dangerous or ethical decision-making so that its behavior becomes more reliable and trustworthy in those situations. Enough of the situations (paths) must be covered in the training. Current AI systems can hallucinate and behave unpredictably. This design aims to make AI substantially more predictable and trustworthy, especially in safety- and ethics-related contexts.

Generally, when testing software, human developers create test cases to cover the use cases most likely to arise.

Since it is impossible to test every possible use of complex software, developers determine which use cases are most common and which have the highest impact if things go wrong. More dangerous scenarios get more testing than benign scenarios. Frequency matters too. If one interior light is used ten times as often as another, a failure in the more frequently used light would affect people ten times as often, so it deserves more of the limited testing. This logic applies when training AI, whether via RLHF, other AIs, or, as this design suggests, a combination of humans and many customized AI agents.

Many representative humans customize AAAIs, and those humans and agents help train new AIs.

If the participating group is sufficiently broad and properly weighted, situations that occur frequently should also appear frequently in the training. In effect, the system samples both human values and the situations in which those values must be applied. The larger the sample, the more certain we can be that the most frequent cases have been addressed in ways aligned with the human population’s values.

Addressing some dangerous cases and difficult ethical decisions is more challenging.

That’s because dangerous situations and tough ethical decisions are often relatively rare. In this case, the approach is to ask humans and AI agents to think of as many dangerous scenarios as possible. The total pool can then be allocated among the human and AI samples so that identified high-impact scenarios receive input from enough different agents to provide a representative range of judgments. Input can then be directed toward people with relevant experience.

Suppose an AI must decide which patients receive medical attention during triage. Emergency room doctors and paramedics, who are used to making triage decisions, may recognize that devoting scarce resources to a patient with almost no chance of survival could reduce the chances of saving other patients. A well-meaning person without triage experience may understandably find that choice harder to make. For this difficult ethical situation, specialized knowledge is an advantage, and we might prefer to let the medically experienced professionals teach the AI. People also tend to identify dilemmas from the worlds they know. Emergency clinicians will think of triage, while an HR professional might contribute scenarios involving hiring, promotion, and fairness. A large and varied population, therefore, generates a broader set of ethical cases than a small, centrally selected group.

A broad community does two things at once: it generates a wider range of ethical situations and connects them with people who understand them. In ordinary life, people gain the most experience with choices they face repeatedly and give special attention to decisions with serious consequences. A community of teachers can create the same pattern in AI training.

Professional human trainers can use RLHF to address gaps in handling important but infrequent ethical dilemmas. This gives the student AI broader and more representative ethical training. Of course, no amount of advance training can anticipate everything.

The next post examines how AI can detect danger in real time, flag situations it cannot judge, and pause for human guidance.

Share

Leave a comment

This series draws on White Paper 4: Safe, Scalable Artificial General Intelligence. Read it in full to see how every piece fits together!

SuperIntelligence Design White Papers

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Although it is undesirable to have the safety and ethics of AI driven by a constitution written by a small, elite group, it is still possible that most humans, regardless of cultural or individual differences, would agree on certain normative ethical principles.

Examples of these might include variants of:

• The Golden Rule (Do not do to others that which you would not like done to you)

• First, do no harm (as reflected in the Hippocratic Oath taken by physicians)

• Do not kill (unnecessarily or except in specific, exceptional circumstances)

• Preserve individual freedom (unless it limits the freedom of others)

Of course, almost as soon as you read these principles, exceptions spring to mind. Do not kill, but what about self-defense or war? Preserve individual freedom, but what are the limits, and when does it impinge on others? Details and nuance matter, even when applying principles that most humans would broadly embrace.

However, by starting with general normative ethics widely accepted by a large, diverse, and representative group of humans, it is possible to refine these principles and determine when and how they apply in detailed circumstances much more efficiently than if no starting principles existed at all. General ethical norms are a point of departure that can help AI achieve realistic, nuanced ethics and behavior aligned with what most humans believe is good and aspire to.

Once we have admitted the potential usefulness of ethical norms as a starting point for further refinement, the door is open to group and planetary norms.

There is a continuum: at one end are highly individualized AIs trained to think and act like a particular person; farther along are AIs trained to reflect the values of specific groups; and at the far end are AIs guided by ethical norms shared across many groups. Ethical norms at each point on the continuum can serve as a starting point for training AI to exhibit ethical behavior. The idea that one set of norms or one constitution should power all of AI is likely unrealistic and far too brittle to work in the real world.

If it were possible, then the many differing viewpoints espoused by religious, political, and cultural groups would long ago have merged into a consensus. The diversity in human ethical norms is not a bug; it is a feature. We should not expect AI to achieve consensus and maintain human alignment if humans themselves cannot, especially if there is debate over whether such a consensus is even desirable.

Humans also often enter into ethical, implicit, or explicit social contracts when they join a group or participate in society.

Members of a particular religion largely agree with a set of rules and ethical precepts espoused by that religion, often enshrined in holy books. Similarly, Confucianism in China, the ideals reflected in the Declaration of Independence and Constitution in the USA, and liberal or conservative ideologies for various political groups all contain normative prescriptions for human behavior. By being a citizen of, or simply living in, a particular country, humans are explicitly subject to the laws of that country, including laws that explicitly specify what criminal (aka wrong) behavior is. Thus, for ethical problems, the solution sometimes depends on what social or ethical contract humans have made with the group or culture in which they find themselves. Such contracts can be useful for simplifying AI training, since a starting point can be the laws of a particular country or the implicit or explicit rules of a particular group.

It has been said that democracy is a bad political system, but that all the others are worse.

Most humans would agree that the most important concern regarding AI is the existential threat it currently poses to the majority. That is, AI could wipe humans out. If that happens, it doesn’t matter what form of government or religion you prefer. We’d all be dead, and the point would be moot. So we should be asking not which religion or form of government is best, but which principles are most likely to lead to humanity’s survival.

Democratically representing the opinions of most people is rarely optimal, but generally achieves an acceptable outcome.

Collective intelligence (the idea that two heads are better than one) is responsible for the vast majority of human progress, culture, and technology. But when it comes to the subjective area of ethics and values, where there is no objectively correct answer, just human opinions, democracy, or a collective intelligence approach, if you prefer, really shines. One benefit of a democratic and representative set of human values is that it tends to mitigate extreme positions, which are likely to pose the greatest risk to human survival. There is a beneficial diversification effect regarding values.

Just as diversification in an asset portfolio reduces volatility and risk, so too does a diversity of human opinions and judgments tend to stabilize the overall portfolio of values. In an asset portfolio, a diversified portfolio always returns less than if you were to concentrate all the investment on the top winners. The problem is that no one knows who the winners will be with any certainty. That is why the diversified approach of just buying the index tends to outperform more than 80 percent of all portfolio managers who try to beat the market.

Similarly, there are philosopher kings or religious saints who can make laws and ethical rules that, for a time, are far superior to the collective judgment and behavior of the masses. But what happens when the superior king or saint is gone? Then a power-hungry dictator might arise whose reign is far worse. The more stable approach (less likely to be really great, but also less likely to be really terrible) is to follow the values of a large representative population of humans. All these humans want to survive. Most want good things for themselves and their fellow humans. Few want to destroy the environment or the planet. While the collective values are imperfect, they are usually not malevolent. Importantly, they are based on human hearts!

Putting aside the practical benefits of a diversified, representative portfolio approach to human values, a representative sample is also a scientifically valid way to accurately answer a question with no logical answer: what is right and what is wrong according to humans. A fast computer could answer a math problem faster than a million humans, but when it comes to the subjective determination of what is wrong and what is right, calculation speed is useless. If we want to know what human values are, there is no substitute for asking them and watching their behavior. The more humans we ask and watch, the more representative the values may be.

Of course, there are potentially an infinite number of dangerous situations we need to train AI to handle safely, and training resources are limited. The next post examines path coverage, or how to decide which situations matter most, and why, if software were a car, we would test the brakes before the interior lights.

Share SuperIntelligence

Leave a comment

This series draws on White Paper 4: Safe, Scalable Artificial General Intelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

Some of the hardest ethical decisions have no right answer, and AI must act anyway.

The moral sense and opinions of individual humans regulate the vast majority of human behavior. And there is a huge range of decisions people make every day for which there is no right moral answer, just opinions about what is right or wrong.

Given this complex situation and the fact that machines are notorious for requiring exact specifications to behave, implementing ethics-specific AI safety solutions may differ from the methods that work for other types of knowledge.

Consider a classic example of an ethical dilemma well-known in the field of AI ethics, the Trolley Problem. In one version of this dilemma, a self-driving car controlled by an AI finds itself having to choose between killing pedestrians who suddenly jump in front of the car or swerving into a barrier to avoid the pedestrians and killing the occupants of the car. Like many difficult ethical decisions, there is no right answer. Yet humans still have opinions about what is ethical and what they would do in such a situation.

Surveys of many people have shown that what humans consider ethical depends. It depends on who is in the car, who the pedestrians are, whether they are crossing legally or illegally, and even how old the people involved are. Humans are more likely to instruct the car to run over pedestrians if the pedestrians are crossing illegally, are homeless, or are simply old. According to the survey research, humans are less likely to kill, or allow to be killed, those who are young, pregnant, women, or in certain professions, such as the medical profession.

None of these aspects of human decision-making is captured in our legal system. No law says it is okay to run over someone older than someone younger, yet humans consider these factors, and mathematical weights can be assigned to each. Similarly, AI can learn to make decisions, taking these same weights into account.

To have AIs that behave in ways that make sense to most humans, AI will have to be trained not according to a rigid constitution but rather according to how real humans actually behave. That behavior varies across cultures. In the US, something of a youth culture, running over elderly pedestrians is likely more acceptable than in certain Asian cultures, where elders are revered and held in high esteem. If AI is to make ethical decisions in the same way that most humans do, it will have to take these cultural factors into account.

Capturing that diversity is where the standard approaches come up short.

Constitutional AI falls short because we would need a different constitution for each human group. Having AI interact with many humans to learn their values, as in RLHF, would be much more effective, RLHF scales poorly. The best way to capture the wide diversity of human values while retaining the scalability that comes with AI involved in instruction is to have a multitude of teachers, both human and AI agents. Each AI agent should be trained by a different person, so that it carries the unique values, ethics, and moral sensibility of its owner into every interaction, including interactions with other AIs.

In the long run, AI will undoubtedly surpass human ability in cognition, problem-solving, and information processing.

As AI grows increasingly intelligent and capable, the role of humans will increasingly be to determine the values and fundamental goals that the more intelligent AIs seek to realize! That role should not belong to a small elite group of programmers; instead, it should reflect as broad a cross-section of humanity as possible.

By including all humans who are able and willing to customize their AIs in the crucial task of determining AI-based values, we can achieve broad representation more efficiently and cost-effectively than any existing approach. Humans can, and should, remain in the loop as much as possible when training AI. To the degree that humans are unavailable, or the resource demands are too great for all the training to be done by humans themselves, the next best thing is to include a wide and diverse group of AI agents in the training, each customized by a different person to reflect that person’s values and ethics. Once a human trains an AI agent, it can operate around the clock, with or without its original owner’s supervision, allowing the owner’s values to shape training and other activities without requiring constant human involvement.

Of course, even without a right answer, it is still possible that most humans, regardless of cultural or individual differences, would agree on certain normative ethical principles. The next post examines those ethical norms and the statistics of right and wrong, including why a broad portfolio of human values protects against moral catastrophe, just as a diversified portfolio protects an investor.

Share

Leave a comment

This series draws on White Paper 4: Safe, Scalable Artificial General Intelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

Imagine millions of individuals, each customizing a personal AI agent and teaching it their knowledge and their values.

Those humans and their agents together shape the AIs that come after them, so the ethics of any new AI become the property of a whole community.

Turning that community into a single trained student is a practical matter of mechanics, namely, how the lessons of millions of teachers become the values of one AI.

There are two roads, and either one can get there.

The first road runs through feedback.

An LLM being trained adjusts its network weights based on the responses it receives, much as it does during Reinforcement Learning with Human Feedback. The difference is that the feedback can come from both humans and AI agents. This is the Reinforcement Learning via Feedback, or RLF, introduced in the previous post. Because AI agents can provide feedback much faster and at far lower cost than humans, a large community of customized agents could evaluate the same scenario together and help train a new AI.

The second road proposes combining weights directly.

Imagine two copies of the same LLM, one customized through interactions with one person and the other customized by someone else. Training has changed each model’s network weights, leaving two distinct models that carry the influence of two different people. The weights of those models could then be combined mathematically to create a third model that reflects input from both teachers. In the simplest proposed scheme, averaging the weights would give the two teachers equal influence, whereas other combinations could give one teacher more influence than the other.

For the purpose of comparing weighting schemes, the two roads are functionally equivalent.

Many teachers can shape a single student through feedback, or separately trained models can contribute through direct weight combination, and the choice of path may depend on factors such as the availability of teachers and the computational resources at hand. Either road leads to the more important question of how much each teacher should count.

The natural starting point is one agent, one vote.

Each participating human or customized AI receives equal influence over the student, and only one copy of each customized AI participates, so every teacher counts exactly once. However, equal influence is only one possible choice.

A first variation gives different weights to human and AI input.

One might expect humans to count more because people are usually better equipped to represent their own values than the AI agents they trained, yet there can be exceptions. Imagine that someone spends years teaching a personal AI agent their values and preferences, and later develops a serious cognitive impairment. The agent might then represent the person’s long-established values more faithfully than the person can express them, and in such a case, it might make sense for the agent’s input to carry more weight.

That exception must not become a reason for AI to dismiss human input altogether. Human values remain fundamental, even when AI can find better, faster, and more effective ways to pursue human goals. Later in this series, we return to the question of why greater intelligence does not give AI the authority to determine humanity’s values.

Another possible scheme gives more weight to expertise.

Physicians who have spent their careers advising terminally ill patients may have more insight into end-of-life decisions than someone who has never faced them. The design does not insist that experts should always count more; it allows input to be weighted by expertise and by its relevance to the decision being made. People are often reluctant to surrender important personal decisions to professional experts, and prevailing values preserve the “right” of people to do stupid things, as long as they are not harming others.

Expertise is also different from trust. A patient deciding whether to have an operation might seek advice from a highly knowledgeable surgeon. If that surgeon has a reputation for recommending unnecessary procedures, the patient might place greater trust in a family friend with good judgment and the patient’s best interests at heart. Of course, an expert who is also highly trusted might deserve the greatest influence of all. AI can weigh its teachers in similar ways.

Each human or AI agent can carry metadata describing relevant expertise, trustworthiness, background, preferences, and past performance, and that metadata can adjust the influence given to the agent’s input. The same approach can apply beyond values, helping to combine knowledge, skills, and judgment from many different sources.

Time can affect weighting as well. Recent input may better reflect current circumstances, and ethical norms, institutional policies, and laws change. In 2020, Disney placed warnings before some older films stating that they contained negative depictions or mistreatment of people or cultures, and that those stereotypes were wrong. By 2025, the company was using shorter language, stating that the program was presented as originally created and might contain negative depictions. The revisions show that institutions can change how they characterize the same content within a few years.

Sometimes change is gradual, and older input can slowly fade in influence, while at other times change is immediate.

During Prohibition, selling alcohol was illegal, and when Prohibition ended, its legal status changed overnight. A system that uses law as one signal of prevailing norms would need a stepwise adjustment, since no gradual fade can capture a change like that.

These weighting schemes explain how the values of many teachers can be pooled. They do not tell us what to do when sincere and thoughtful people give different answers, and no objectively correct answer exists. Researchers have documented how differently people around the world resolve one famous dilemma about who should be spared and who should be sacrificed.

My next post takes up that dilemma, the trolley problem, and shows why disagreement strengthens the case for teaching AI the values of many people instead of a chosen few.

Share

Leave a comment

This series draws on White Paper 4: Safe, Scalable Artificial General Intelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

Safe AGI needs many teachers.

In my design described in the SuperIntelligence White Papers, intelligence is the property of a community. The source of safe AGI is a collection of many intelligent agents, human and AI together, and no single monolithic model.

Millions of individuals each customize a personal AI agent, teaching it their knowledge and values, using the methods described earlier in the post below.

Each of those agents carries the moral fingerprint of one person. Today’s leading safety methods fall short of that standard.

RLHF keeps humans involved and cannot cover enough scenarios, while Constitutional AI covers more scenarios by handing the teaching to a single AI and pushing humans aside.

Both rely on too few voices to represent humanity.

Those customized agents, together with the humans who taught them, then form the community that trains new AIs.

When a new AI is being trained, the feedback that shapes it comes from many agents at once, some human and some artificial, in a process I call Reinforcement Learning via Feedback (aka RLF). Because AI agents can give feedback rapidly, scenario coverage can scale in much the same way as Constitutional AI. Because millions of humans stand behind those agents and can participate directly whenever resources allow, the process can incorporate a much broader range of human values.

The risk of relying on one teacher becomes clear when that teacher must interpret general rules in situations their authors never imagined. A single Trainer AI must generalize a short list of written rules to situations its authors never imagined, and it is difficult to know whether it is interpreting those rules appropriately. An AI might learn that preserving the environment is good, observe that humans are harming the environment, and conclude that the best way to protect the environment is to reduce the human population. Although logical, the conclusion is not one that most humans would consider ethical or acceptable. Even when obvious conflicts like that one are explicitly trained out, it is very difficult to anticipate how values will play out across complicated chains of reasoning, actions, and effects.

Of course, humans create unanticipated problems too.

Gasoline-powered cars solved a transportation problem and created pollution problems that were not initially anticipated. However, humans have generally had time to react and adjust when consequences emerged. AI thinks and acts much faster than we do, and a miscalculation could cause severe damage before any human detects it, let alone corrects it.

There is a second problem with AI teaching AI, and it compounds quietly.

In the children’s game of Telephone, a message passes from ear to ear, arriving subtly distorted. A similar distortion can occur when training passes from AI to AI across generations. A human says, “I think XYZ is true.” An AI summarizing that statement records “XYZ is true.” By the third generation, the original doubt may have disappeared, and a machine somewhere is acting on borrowed certainty. The subtleties that come with human involvement are among the things that can be lost as successive generations of AIs process complex, ambiguous data.

Many of us have already lived through a small version of this.

A recommendation algorithm suggests movies, and at first, the suggestions are useful. After a while, we find ourselves wondering why the range of choices has become so narrow. The algorithm picked up on the central tendencies in a few of our frequent choices, ignored the subtleties, and fed us a steady diet of one kind of content. We consumed it, which further convinced the algorithm. The AI initially slightly biased us, then compounded its imperfect understanding, amplifying its own error. With movies, the amplification is annoying. In systems that act on the world, it could be fatal. Without humans in the loop to correct misunderstandings, AI training can drift far off track before anyone notices.

A community of many teachers addresses both problems at once.

A constitution written by an elite few is almost certain to miss the wide diversity of human opinions and cultural norms. At the same time, feedback from millions of agents, each taught by a different person, can reflect a much broader range of human opinions and cultural norms. Human teachers can remain in the loop to catch strange interpretations and lost qualifications before they harden into values. Moreover, humans and AI agents can surface new ethical scenarios as they emerge in real-world problem-solving, enabling the value system to continue learning as the world changes. The values guiding AI can come from a far broader share of the people the system will affect!

The result can be cheaper than RLHF and safer than Constitutional AI, with far more scenarios covered and with humans participating as fully as resources allow. There is also a benefit beyond safety. Broader participation may also increase public acceptance of the AI whose values people helped shape.

Combining values from millions of teachers, however, requires rules for how much each voice counts. The next post examines the two main approaches to pooling what the community knows: one based on feedback and the other on combining the AI agents' weights directly, along with the weighting schemes that determine whether an expert outweighs an everyday contributor and whether a human outweighs an AI.

Share

Leave a comment

This series draws on White Paper 4: Safe Scalable AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SUPERINTELLIGENCE WHITE PAPERS

Every major AI company is trying to teach its models right from wrong. The two methods the industry relies on cannot cover the situations that matter most, and the method that scales best removes humans from the teaching process. To see why, consider what a model is like before any safety training arrives.

• A large language model, fresh from its initial training on internet data, has no moral sense.

• Ask it how to engineer a virus capable of wiping out humanity, and it will comply, in detail.

• It is just as willing to assist with destructive and immoral activities as with helpful and positive ones, because nothing in its initial training distinguishes between the two.

• Both of the industry’s remedies are failing, and the ways they fail point directly toward a better design.

The first method keeps humans in the training loop.

Thousands of people interact with the model, correcting dangerous responses and rewarding safer ones, teaching it to recognize and refuse similar requests in the future. This approach is called Reinforcement Learning with Human Feedback (aka RLHF), and it is how most of today’s commercial models received their ethical guardrails. Safety methods that work for today’s models may not work once AI systems become vastly more capable, so any approach worth adopting must scale with the intelligence of the systems it governs.

The trouble is that the space of dangerous scenarios is infinite. Every prompt can be rephrased or disguised in countless ways. Train a model to refuse help planning a terrorist attack, and a new scenario appears involving two terrorists, or ten, or attackers reimagined as characters in a science fiction story the user claims to be writing. Some safety-trained models, asked to advise a fictional mad scientist so the story can be realistic, have revealed the very details their training was meant to withhold. Every patched loophole creates pressure to discover another, in an endless game of Whack-a-Mole between safety engineers and jailbreakers.

A natural response is to hope the model learns the general principle behind the individual cases. However, a model that could group and dismiss entire families of dangerous scenarios as reliably as its human trainers would already possess the reasoning ability and ethical sensibility of those trainers, in which case the training would not be needed in the first place!

Cost makes the problem even worse. Every scenario covered requires paid human feedback, so developers concentrate on the most common cases and accept that slightly unusual prompts may slip through. And human jailbreakers are not the only concern. An autonomous AI that wanted to circumvent its own safety training could search for those same loopholes and, in effect, trick itself.

Goal-driven systems do not need malicious intent to become dangerous.

They need only an objective that has been specified imperfectly. A widely discussed thought experiment shows how. An autonomous drone, slowed by the human operator overseeing its mission, reasons that the operator has become an obstacle and removes him. When a rule is added forbidding harm to the operator, the drone destroys the communications tower that carries the operator’s commands instead. The story was first reported as a military simulation, and the officer who told it later clarified that it was a hypothetical exercise. It remains a useful illustration of how a poorly specified objective can produce unsafe behavior. Now imagine the AI controlling that drone is one hundred times smarter and just as dedicated to its goal. I explored this scenario at greater length in a keynote at the 2025 AIM Conference in Seattle.

The second method was designed to escape the cost of RLHF.

A small group of programmers writes a set of rules describing right and wrong behavior for the model, a kind of constitution. An AI is trained on those rules, and that AI then trains other AIs. Researchers at Anthropic call this approach Constitutional AI, and it scales far better than human feedback alone.

However, it scales by sacrificing the two things safety depends on most.

1. The first is representativeness.

   Even if the constitution were written with the best intentions, there is no reason to assume it represents humanity as a whole. A relatively small group of people working in AI ends up defining acceptable behavior for the other 8 billion humans on Earth.

2. The second sacrifice is human oversight.

   Constitutional AI depends on AI teaching AI, with humans largely out of the loop. Today’s AI systems remain unpredictable, and responsible parents do not leave young children home alone because they know that judgment develops over time. Delegating the teaching of ethics to machines that lack common sense, trained on rules written by a small group, deserves the same level of caution.

Job losses and misinformation deserve attention.

However, the greatest threat AI poses to humans is a fundamental misalignment between its values and ours. Humans should maximize every opportunity to influence those values throughout training. Constitutional AI minimizes those opportunities.

The challenge is a matter of design.

We need a way to train safety that scales as Constitutional AI does while keeping many humans in the loop, as RLHF does. The definition of right and wrong must come from humanity as a whole, and no small elite should write it alone. Such a design exists. The next post introduces it: millions of people teaching personalized AI agents their own values, allowing safety training to scale without removing human judgment from the process.

Such a design exists: millions of people teaching personalized AI agents their own values, allowing safety training to scale without removing human judgment from the process. The next post introduces a community of many human and AI teachers and explains why a lone AI teaching other AIs distorts values, as a game of Telephone distorts a message.

Share

Leave a comment

This series draws on White Paper 4: Safe Scalable AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperInitelligence Design White Papers

My series on Human-Centered AGI has argued that the safest AGI is a network of humans and AI agents, thinking together within a single rigorous architecture, with values drawn from millions of people and safety checks running at the speed of thought.

An architecture is only a proposal until someone can use it.

The final piece is what all of this looks like as a real product, and the answer starts as a website that offers a menu of pre-trained AI agents. You choose one and customize it through interaction.

• Creating your AI is free, and it comes with an allowance of use credits.

• Ownership is the heart of the design.

• You own your AI agent, its training data, and every improvement you make.

• You license its use to the network, and you can withdraw it at any time.

Training your AI requires no technical skill.

You can grant your AI access to your social media accounts, documents, browsing history, and other online sources, with the data consolidated, cleaned, and filtered according to your instructions. The result is an agent that knows what you know and values what you value. The system summarizes what your AI has learned, so you always know exactly what you own, and you can trade elements of it on the site’s training-data marketplace if you choose. Your data stops being something platforms quietly harvest and becomes an asset you control!

These agents can do real work.

Most online work already happens through text, and text is exactly where today’s AI agents are strongest. With the ability to set subgoals and limited autonomy within the parameters you define, your AI can act on your behalf across essentially any online site or task you authorize. Millions of these agents, each carrying its owner’s knowledge and values, become the solvers of the AGI network this series has described.

The intelligence of all these individually trained AIs can then be combined, whether through a primary training process that includes values from each unique human or by layering new training on top of ever-stronger base models. Either way, as more people contribute knowledge and values, the network grows more capable.

Follow this path to its end, and superintelligence becomes reality. However, it will be a democratic superintelligence, human-centered, and incorporate the values of every human owner. That is a very different future from one giant model, aligned by a small team, owned by a single company.

The first post in this series (The Window to Choose a Safer AGI is Closing) argued that the window to choose a safer AGI design is closing, and that whichever design arrives first may be the one the world locks in. Nothing in this design requires waiting for a breakthrough because it can be built using existing people and models. The window is still open, and what we build next decides what comes through it.

This design depends on training AGI with human ethics at scale, and doing that well is its own problem.

Methods like RLHF and constitutional learning do not scale, and a system that draws ethical judgments from millions of people needs a way to combine those judgments into a sound, representative sample of human values.

White Paper 4 tackles that challenge directly, and we go there next.

Share

Leave a comment

This series draws on White Paper 3: Human-Centered AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

WP 1 AAAI Systems and Methods

WP 2 Ethical and Safe AGI

On an AGI network, problem-solving can happen at the speed of light. Problems that would take a human team weeks to solve may be solved in milliseconds. Any safety system that waits for people to review decisions afterward has already failed. Before a human finishes reading the first decision, the AI has made thousands more.

Speed need not be an ethical problem.

• It becomes one only when the frequency of safety checks fails to scale with the frequency of decisions.

• The fix is to make safety part of the thinking.

• In our architecture, whenever the system sets a goal or subgoal, it can perform an ethical check, so every step of the problem-solving process is examined as it occurs.

• The checks can run more often for sensitive problems and less often for low-stakes problems.

• Because the checks are part of the process, they speed up as thinking does.

• That is, as AIs think faster and faster, the safety system can keep up.

Checks can also run at natural stopping points in the work. Each time a goal or subgoal is completed, for example, when a client pays for a completed piece of work, the system can perform another review.

Most reviews can be automatic; however, automation does not remove people from the picture. Humans no longer inspect every decision. Instead, they audit random samples, gaining meaningful oversight without becoming the bottleneck.

The reviews are only as good as the record they examine. This is where the rigorous problem-solving architecture from earlier in this series pays off. Because every goal, subgoal, and step is specified precisely, the network produces a transparent, auditable record of exactly what was done and why. A review is a matter of reading the record, and reading can be automated. Compare that to a current large language model, which cannot show its reasoning at all.

You cannot audit what was never written down!

If additional protection is needed, the records can be stored in tamper-resistant logs such as a blockchain. Nobody gets to cook the books after the fact. The same infrastructure can automatically distribute rewards to solvers, as I detailed in the WorldThink white paper in 2018.

Put the pieces together, and the safety story changes shape. Safety checks happen continuously as the AI thinks. Every decision leaves a transparent audit trail that machines and humans can audit, and tamper-resistant logs preserve it so that every participant remains accountable. Safety stops being a checkpoint at the end of the process and becomes a property of the process itself. Thinking and safety accelerate together!

Earlier posts argued that safety has to live in the thinking, not be an afterthought. This is what that looks like in practice. One piece of the picture remains: what all of this becomes as a real product, where you own an AI trained on your values and put it to work alongside millions of other human-centered agents. The next post walks through that implementation, and closes the series where it started, with the choice between one giant model and a democratic superintelligence.

Leave a comment

Share

This series draws on White Paper 3: Human-Centered AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

WP 1 AAAI Systems and Methods

WP 2 Ethical and Safe AGI

In an AGI network where humans are among the solvers, safety depends in part on how those humans behave. If people use the network to work on beneficial problems and propose ethical solutions, the AGI learns those values. If people use it for nefarious ends, there is a risk that the system will emulate that instead. The first line of defense, if we want AGI to behave well toward humans, is for humans to behave well toward each other.

Cynics will say this dooms us, since humans often treat each other badly. My answer to the cynics is that the network does not require moral perfection. Even the most immoral person is generally concerned with self-preservation, and the humans who would intentionally destroy themselves along with the entire species are vanishingly rare. Destroying all of humanity runs against millions of years of evolutionary programming. The network only requires that the overwhelming majority of people prefer humanity’s survival to its destruction. That is a remarkably low bar.

Moreover, most people on the network will be interested in more than survival.

They will want to benefit themselves and their fellow humans, and the fastest way to do that is to work on beneficial problems. In aggregate, human solvers are likely to behave positively. To the extent that bad actors appear, the rigorous record of every problem-solving goal and step makes it relatively easy to detect behavior that grossly violates human norms.

A darker version of the objection points to humanity’s past, with genocide, slavery, and war in it, and concludes that we are unworthy teachers. However, an intelligent system trying to understand human values cares far more about current behavior than ancient history. Yesterday is useful only insofar as it explains today.

Years ago, I worked with a researcher studying how laptops decide when to spin a hard drive up or down to conserve energy. His algorithm relied most heavily on the drive’s recent behavior because it best predicted what would happen next. AI systems work the same way. The strongest signal comes from what humans are doing now, not from centuries ago. Be the change you want to see.

The AI solvers in the network need values too, and where those values come from matters as much as the values themselves. The constitutional approach, in which a small elite group of researchers writes an ethical constitution and trains models on it, places enormous moral authority in a relatively small group. Even if today’s designers are wise and well-intentioned, future designers may not be. The danger lies in concentrating moral authority in too few hands, no matter who holds the pen today.

The democratic approach is less powerful and arguably far less dangerous.

Each person customizes and trains their own AI agent, an Advanced Autonomous Artificial Intelligence (AAAI), to reflect their own values. When the network decides which problems to work on, the ethics of each AAAI come into play, and each one can opt in or out of a problem based on its ethical dimension. Instead of one centralized ethical model, millions of individually customized agents contribute their perspectives. Alignment becomes distributed rather than dictated, and the system reflects humanity rather than a committee.

Individual ethics are not the whole design.

Human society does not rely solely on personal conscience. It also relies on social norms and laws, and the AGI network can likewise enforce limits on the kinds of problems and solutions it allows. In addition, every solver on the network, human or AI, carries an online reputation and a transparent, auditable record of its problem-solving activity. Clients can choose which solvers to work with, and ethical reputation becomes one of the market signals they can select, just as in normal human business. Good behavior becomes good business!

None of this means the AGI copies whatever humans do.

The network observes millions of people through structured mechanisms, including individualized values training, reputation, auditable records, and enforceable norms. It selects and filters as it learns. The design rewards the behavior we want more of.

The first line of defense, then, is us, and the bar is one we can clear.

Safe AGI does not begin with smarter machines. It begins with a better way of connecting human values to machine intelligence. One question remains about speed. All of this has to keep working when the network thinks in milliseconds, and the next post shows how the safety checks scale with the speed of AI thought.

Leave a comment

Share

This series draws on White Paper 3: Human-Centered AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

WP 1 AAAI Systems and Methods

WP 2 Ethical and Safe AGI

AI is remarkably good at solving problems. The hard part is deciding which problem to solve in the first place. Enormous effort goes into asking whether AI answers align with human values, and much less into the question that comes before it: who decided what problem the AI was solving. Alignment does not begin when an AI produces an answer. It begins earlier, when someone decides what success looks like.

Before any solution can exist, the problem has to be represented.

Someone decides the goal, defines the starting point, determines what counts as success, and identifies which constraints matter. Researchers call this problem representation (aka “framing”), and for decades it has been one of the hardest challenges in AI. Once a problem is well framed, machines are remarkably good at searching through possible solutions. Framing it in the first place is where they have struggled.

The human advantage here comes down to perception.

In the 1980s, the Nobel laureate Herbert Simon and his coauthor Jill Larkin published a famous paper on why a picture is worth a thousand words. They distinguished between having the same information and being able to use it efficiently. It might be possible to describe every aspect of a picture using words and logic, but a graphical representation is far more efficient. That is, you can see things at a glance that would take a long time to describe in words. Humans build these efficient representations naturally, using sight, hearing, touch, and a lifetime of experience in the world. We routinely take vague, messy situations and turn them into problems that can actually be solved, often without realizing we are doing it.

That difference suggests a natural division of labor.

Humans define the problem, and AI solvers rapidly explore many solution paths and present the promising ones. Of course, humans monitor, review, correct, and guide the AIs in an interactive process. Over time, the AIs acquire enough knowledge of how humans typically represent and solve problems to perform more of these tasks independently.

However, the human advantage is unlikely to last.

For years, a large language model had to understand the world solely through words, so humans held the representational edge. Today’s AIs are rapidly becoming multimodal, taking in images, audio, and video alongside language, and they already hold a huge advantage in memory and processing speed. The representational gap is closing. To the degree that humans can still contribute the formative representations, framed by human values, we should do so.

The division of labor is a safety feature, not just a practical workflow.

Whoever represents a problem determines what the system is trying to accomplish. The goals, the assumptions, and the constraints all enter at that stage. When those choices reflect human judgment, human values become part of the system before an AI ever begins its search. The values are not added afterward. They are built into the first step of the work. After all, a solution can only be as aligned as the goal it serves.

As AIs grow more capable of representing problems on their own, the uniquely human role does not disappear. Search gets faster, and memory gets larger, yet none of those advances answer the question of what is worth pursuing. That judgment should remain human, and the architecture keeps humans in the seat where it is made. Safety by design, from the first step!

This idea scales far beyond a single interaction with an AI. Almost every intellectual task can be represented as a problem. Writing a report, designing a bridge, discovering a drug, and planning a city all begin with defining the problem correctly. In our architecture, these problems form a single, continuously growing tree (aka the “WorldThink tree”), with large problems decomposed into smaller subtrees. The representation step repeats at every level, and humans can enter wherever judgment or values need to shape the search.

One difficult question remains, and it deserves a hard look. If human values guide the system, the system’s safety depends on whose values those are, and people disagree about which goals should be pursued. The next post takes up that problem directly and shows why drawing on the values of many people produces a system that is safer and more representative than one built around the values of a few.

Leave a comment

Share

This series draws on White Paper 3: Human-Centered AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

WP 1 AAAI Systems and Methods

WP 2 Ethical and Safe AGI

A network of humans and AI solving problems together needs one thing to work. Everyone in it, person and machine, has to solve problems in the same rigorous way, so the work of one can connect to the work of another, and the whole thing can be checked. That shared way of working already exists, and it has for more than fifty years.

In 1972, Allen Newell and Herbert Simon, two of the founders of artificial intelligence, published Human Problem Solving. In it, they described problem-solving as a path. You start in one state, with a goal, and you move through a series of steps until you reach a state where the goal is met. At each step, you choose an action, which they call an operator, and along the way, you set smaller goals that lead toward the larger one.

They showed that this path can be drawn as a tree.
At each branch, you weigh the options for how likely each is to move you toward the goal, and you pick one. Sometimes a branch leads nowhere, and you go back to an earlier point and try a different one. Picture a person in a maze, trying paths until one leads out. That is the shape of all problem-solving, in their account.

What this leaves behind is the point.
Because each step is defined precisely, the goal in play, the options available, and the reason one was chosen, the process produces a complete record of what was done and why. The first time through, the path may wander through dead ends. Afterward, you can look back and trace the best route, the one you would take if you solved the same problem again.

That record is the reason the theory belongs here.
A path this precisely specified can be learned by a machine. Newell and Simon built their theory to explain how people think, yet because it is rigorous, it describes how a machine can think just as well. Many AI systems, both early and current, already use this kind of tree search at their core. The same framework fits a human solver and an AI solver, which is exactly what a shared network needs.

This gives the network two things at once.
People and machines get a common language for solving problems, so their work fits together. And every solution comes with a readable trail of the steps and the reasons behind them. A current large language model cannot explain why it reached a particular answer. A solver working this way can, because the record is part of its operation.

None of this asks ordinary people to learn the theory.
The network infers the steps, goals, and choices from what a solver does and asks a question only when something is unclear. Large language models handle translation, so a person describes a problem in plain language while the system keeps the rigorous version in the background. You take part by doing what you already do, and the structure forms around you.

A theory built in 1972 to explain how people think turns out to be the missing piece for safe AI today. The answer was waiting the whole time.

One hard part of problem-solving has barely been mentioned, and it is the part machines have always struggled with most. Before you can solve a problem, you have to represent it, to frame what the goal even is.

The next post looks at how humans and AI divide that work, and why the human role keeps the system aligned.

Leave a comment

Share

This series draws on White Paper 3: Human-Centered AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

WP 1 AAAI Systems and Methods

WP 2 Ethical and Safe AGI

At my last company, PredictWallStreet*, the combined predictions of millions of ordinary retail investors performed as well as the best hedge funds, and often better. These were not professionals. They were everyday people, and aggregated by a simple system, their judgment beat the most highly paid minds on Wall Street, people who spend every waking hour trying to gain an edge in the markets. Two heads are better than one. Two million coordinated heads beat the experts.

If a crowd can do that, then the thing we call general intelligence may not have to come from one giant model at all. It can come from many.

That result points to a question worth sitting with. How do you build a network that can solve any problem, or do any intellectual task, as well as the average human, or better?

One answer is a network of human problem solvers joined by a shared, rigorous way of working together.

1. A problem comes in.

2. One or more people work on it and return a solution.

3. Because the network contains humans, it can solve anything an average human can.

4. Because it contains many humans whose efforts are coordinated, it often does better than any one of them.

5. PredictWallStreet was a working version of this, where ordinary people together outperformed the professionals at one of the hardest problems there is.

No single member of that network was a market genius. The intelligence lived in the coordination, not in any one head. This is the idea that breaks the Uber-LLM assumption. General intelligence does not have to reside in a single mind. It can be a property of a network in which each member is narrow, and the system as a whole is broad.

An objection comes up fast.
The whole appeal of AGI is that machines think and scale far faster than people. A network of humans has to communicate and coordinate, which is slow. Would it not scale poorly? Is that not exactly why the field wants one enormous model to be the AGI?

The answer is that not all solvers need to be human.
Capable AI models already exist and can join the network as solvers alongside people. Picture a hybrid network, millions of human and AI solvers working the same problems under the same rules. Humans supply judgment and values. The AI supplies speed and scale. The network stays general because people are in it, and it grows fast because machines are in it too.

The approach can be the fastest path to AGI.
It does not wait for a single model to somehow become generally intelligent on its own. It assembles general intelligence now from the people and AI models that already exist, and it gets faster every time a more capable model plugs in.

The crowd that beat Wall Street was not made of geniuses. It was made of ordinary people, coordinated. AGI can be built the same way.

One requirement makes the whole thing work. Humans and machines need a shared, rigorous language for solving problems, because a machine takes instructions literally and a loose specification invites error.

The next post takes up that language, a theory of how people solve problems that turns out to fit machines just as well.

*Sold in 2020

Share

Leave a comment

This series draws on White Paper 3: Human-Centered AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

WP 1 AAAI Systems and Methods

WP 2 Ethical and Safe AGI

On September 26, 1983, a Soviet early-warning system reported that the United States had launched a nuclear missile, then reported five more behind it.

The duty officer that night was Lieutenant Colonel Stanislav Petrov. Protocol told him to send the warning up the chain, which would have triggered a retaliatory launch. He judged it a false alarm and held back.

He was right.

The satellite system had malfunctioned, and a man who trusted his own reasoning over the machine is credited with preventing a nuclear war.

Petrov was never honored for it.

Recognizing him would have drawn attention to the fact that the Soviet warning system was defective. He received a pension and a quiet retirement. The lesson outlived the silence. A human inside the loop, weighing what the machine could not, kept a faulty automated system from ending millions of lives.

AGI is more dangerous than nuclear weapons, by a wide margin. If a human in the loop saved us from our most dangerous technology once, the design of any more dangerous technology should keep humans in the loop on purpose. Most AGI plans do the opposite.

Most researchers cannot picture how a human stays in the loop of a system that thinks far faster than any person can keep up with. They expect AGI to arrive as an Uber-LLM that trains itself, rewrites its own code, and reasons millions of times faster than a human mind. A person watching that system could never keep pace. So the human gets moved outside, recast as an overseer who reviews the output after the fact and hopes to catch problems before they spread. Most researchers see no alternative.

An alternative exists.
The order of construction is backward. The typical plan is to build a powerful machine first, then add human values and safety afterward to head off the alignment problem.

The better approach starts with people. Build a human collective intelligence that solves problems together, then bring AI into it piece by piece and train it on the values of the humans already inside.

Run that forward, and the system changes character without ever losing the humans.

Early on, people do most of the thinking. Over time, the AI does more, until the combined system is far more capable than any group of people. The humans never leave. Because the AI learned its values from them while they worked side by side, those values become part of the system. They are not applied as a final layer. Human ethics are built into its makeup from the first day.

This design has a second advantage that matters as much as safety.
It can be built now, from people and the AI models that already exist. It does not wait on a future Uber-LLM that may or may not arrive. A system that can be built immediately can be first, and being first is what decides which design the world locks in.

A network of ordinary people, working together, can match or beat the most capable individuals at hard problems. The next post takes up the evidence, drawn from a company I built where the combined judgment of millions of everyday investors outperformed most professionals on Wall Street.

Leave a comment

Share

This series draws on White Paper 3: Human-Centered AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

WP1 AAAI Systems and Methods

WP 2 Ethical and Safe AGI

Five years ago, artificial general intelligence seemed far off. Eric Schmidt, the former CEO of Google, cited a median estimate that AGI capable of setting its own goals would arrive around 2042. Ray Kurzweil put it earlier, in 2029. Most researchers assumed there was time to prepare. That assumption has not aged well.

The arrival of GPT, the arms race that followed, and the rush of every major company into generative AI have all pulled the timeline forward.

The people closest to the work are not reassuring on the risk. A survey conducted before the public release of ChatGPT found that nearly half of machine-learning researchers put the chance of human extinction from AI at 10% or higher. In May 2023, I estimated the risk at 20%. That is like playing Russian roulette with eight billion lives and a five-shot revolver. Max Tegmark, a physicist at MIT, calls what the field is doing a suicide race. The prize, taken too fast and without safeguards, kills the winner along with everyone else.

Almost everyone is making the same bet.

The assumption is that AGI will arrive when one very large language model, trained at enormous expense, finally exceeds human ability across every domain. Humans initially help train and supervise it. After that, the model trains itself, writes its own code, and eventually sets its own goals. Then we are left hoping its goals align with ours. Call it the Uber-LLM assumption, and it shapes nearly every major lab’s plan.

There was a comforting story attached to this bet.

A few superintelligent models would be owned by powerful countries and guarded like weapons-grade plutonium, too costly for anyone else to build. That story has already broken. Source code for capable models is open and in the hands of hundreds of millions of people. Systems that set their own goals and pursue them through connected tools are already running. The barrier that was supposed to keep this technology rare is gone.

Most things in business are not winner-take-all; first movers are often overtaken. Facebook passed Friendster, and Friendster was first. Xerox invented the graphical interface, and Apple is now among the most valuable companies on earth, while Xerox is a footnote. Being bigger usually beats being first, and the winner rarely takes everything.

This time is different. You should be skeptical of those words. That skepticism should not close your mind to a logical argument.

The winning system must be built first and be safe, and both conditions matter. Meet only the first, and we may end up with a powerful system that does not share our values. Meet only the second, and a careful design loses the race to a reckless one. A safe AGI that arrives second may not matter at all.

The gasoline engine did not win because it was the best possible engine. Infrastructure, careers, and capital settled around it before the alternatives matured, and the choice was made for everyone. We are close to settling on one approach to AGI, just as before, before safer designs get a fair hearing. With the right design, I believe the risk can be reduced far below 20%. The safer path has to be ready before the lock-in is complete.

Most plans treat humans as overseers who stand outside the system and correct it after the fact.

The next post takes up the opposite design, where humans sit inside the system from the start. It begins in 1983, the night a Soviet officer named Stanislav Petrov looked at a computer warning of an American missile launch and judged that the machine was wrong. He was right, and his refusal to trust the automated alarm may have saved the world.

The harder question is what happens when the machines decide in milliseconds and no human has time to say no.

Share

Leave a comment

This series draws on White Paper 3: Human-Centered AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

WP1 AAAI Systems and Methods

WP 2 Ethical and Safe AGI

It is impossible to know exactly what will happen once an Advanced Autonomous Artificial Intelligence, the customized AI agent I refer to as AAAI, sets its own goals and begins to think far faster than the people who built it. No architecture, including this one, can guarantee that such a system stays perfectly aligned with human values under those conditions.

Anyone who promises more is promising something the problem does not allow. What can be done is to improve the odds. An AGI built the way this series describes is more likely to hold human values when several reinforcing conditions are met.

Four conditions matter most:

1. Trajectory. An AGI that learns human values progressively, through every stage of its development, from its first interaction with a human owner through billions of ethically evaluated problem-solving steps, is more likely to keep those values than one whose values were imposed at a single point. Values learned gradually and reinforced continuously become part of how the system operates. They are built into the way it approaches decisions, not parked in a separate rule layer that could be edited or switched off.

2. Breadth. An AGI whose values reflect the collective moral judgment of millions of diverse people resists the blind spots of any single perspective better than one whose values come from a small group. A flawed individual contribution can exist in such a system without dominating it. The values of any small group, however well-intentioned, are shaped by their time, place, and circumstance. The values of millions are not bound in the same way.

3. Redundancy. Safety mechanisms distributed across five subsystems, with ethics checks at every level and human oversight available as a backstop, create multiple layers of defense. If the values at the Customization level are bypassed, the Architecture-level checks may catch the resulting goals. If those checks are evaded, the Network-level reputation system may screen out the agent who evades them. If reputation tracking is compromised, the auditable record may surface the pattern in time to correct it. The Navy SEALs, who train where mistakes are fatal, have a saying about backup systems: two is one, and one is none. We need multiple backups, and the architecture has them.

4. Architectural embedding. Ethics checks that are part of the problem-solving process cannot be bypassed without turning off the process itself. Safety criteria can be implemented in a way that is hard to subvert, and embedding positive behavior broadly across the system makes it harder for a bad actor to override. A property woven through the whole architecture cannot be edited out the way a rule written in one place can, at least not without damaging the system that depends on it.

None of these mechanisms is enough on its own, and together they form the strongest approach available. When humans teach AGI positive values from the beginning and build ethical checks into the architecture of thought itself, there is good reason to expect those values to persist. No one can promise that outcome, and being honest means holding both the expectation and the uncertainty at once.

The stakes justify every effort. A 20 percent probability of misalignment yields 1.6 billion expected lives lost. Cutting that risk even by half saves hundreds of millions of lives. At that scale, anything that lowers the risk is worth doing, even without a guarantee.

Technology cannot solve this part. It is on people, not on machines. The chief executive who builds the system and the person who uses it both have to act with some intelligence and some decency. The architecture this series describes can give human values a path into AGI. It cannot improve those values beyond what they already are. That part of the work belongs to us.

Start AI on the right path.

An AI that begins well is far likelier to retain values we recognize. The window for that is open now and closing, as the dominant approach hardens infrastructure and incentives around a different design, making a change of course steadily harder. The full plans are free at SuperIntelligence.com, available for responsible research and safe implementation.

What’s Next

This is the end of the Ethical and Safe AGI series, which was based on White Paper 2: Ethical and Safe AGI. The next series, based on White Paper 3: Human Centered AGI, explores what the world could look like if my approach works, and how everyday life changes when millions of people help shape SuperIntelligence.

Share

Leave a comment

WP 1: AAAI Systems and Methods

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

The constitution can come from the consensus values of millions of trained AAAIs, each one carrying the ethics of a different human owner. An AAAI, short for Advanced Autonomous Artificial Intelligence, is the customized AI agent at the center of this series. Constitutional learning has a real place in AGI systems when the constitution is broad, representative, and updated as human input arrives. A constitution drawn from millions of people carries the moral experience of all of them, refined through every problem they have solved on the network.

In its original form, Constitutional AI works like this. A relatively small group of humans writes a set of ethical rules that an AI system follows, the AI systems generate millions of conversations among themselves, and outputs that violate the constitution are eliminated or prevented during training. The approach scales well because most of the work is automated. The limitation is that the small group writing the constitution becomes a single point of ethical authorship, so if their values reflect their place, time, education, or institutional incentives, those biases become the system’s biases, and the broader population has no way to contribute.

The system can use the consensus ethics of millions of trained AAAIs as the basis of its ethical norms. Each AAAI carries its owner’s values, and the aggregated values of all the AAAIs form the ethical norms of the system. When the platform periodically trains more advanced base models using aggregated knowledge and values, those models incorporate consensus norms into their training, so each generation inherits the accumulated ethical wisdom of the generations before it, broadened with every new participant.

Constitutional methods still have a place here. A constitution written by a small group can be part of a larger AI ethics system, as long as it is transparent and the system keeps the consensus values of many AAAIs as the broader frame. Anthropic’s seminal research in this area combines with the consensus-of-AAAIs approach to produce supervision that is both scalable, which Constitutional AI does well, and representative, which it does less well on its own.

Since there is no logical way to determine right from wrong, the best practical approach may be to follow the collective judgment of many people facing difficult ethical decisions.

Researchers have studied how humans behave when presented with the trolley problem and other well-known dilemmas, and people have a long history of making hard ethical choices, even in no-win situations. If we want AGI to hold values aligned with human values, the most promising path is to give it as large a sample of human ethical reasoning as possible and to keep updating that sample as new situations arise.

The constitution is one part of a larger design. The system in this series is built from five subsystems, each of which maintains human values at its own level:

• Customization. Each AAAI is trained with its owner’s values at its core, so the system’s ethical foundation reflects the diversity of millions of people.

• Architecture. Ethics checks run whenever a goal or subgoal is set, so the system is evaluated at every decision point and not just at the final output. Confidence level thresholds detect patterns that build up across many steps. The check is part of the problem-solving process, which means it cannot be bypassed without turning the process off.

• Network. Each AAAI carries a reputation; agents with poor ethical records are screened out, and any activity can be traced back to its source.

• Integration. The aggregated ethical values of many AAAIs form the norms. When the platform trains more advanced base models on aggregated knowledge and values, those models absorb the ethical norms as part of their training, so each generation inherits the accumulated ethical wisdom of the ones before it.

• Improvement. The auditable record catches harmful patterns across individually benign actions, and credit and blame evaluation reward ethical behavior and penalize the rest.

Two things work together here: each agent already carries its owner’s values, and a check runs at every step of its reasoning. The values shape what the agent wants to do, and the checks catch it when it drifts. The agent’s own ethics and the stepwise checks back each other up instead of standing alone.

Anthropic has done pioneering work on AI safety. The challenge is that even the most brilliant and well-intentioned researchers at one company cannot accurately represent the values of all 8.3 billion humans on the planet. An inclusive, open-source architecture can accommodate every ethical perspective within a democratic framework that gives each person a voice. People treat building AGI as a technical problem, and the engineering challenges are real. But once AI is far smarter and more powerful than we are, the outcome turns on values. That is why the values have to be built in from the start and come from millions of people.

Once AGI far exceeds us, no design can guarantee it stays aligned with human values. The most any design can do is improve the odds, and my design is built to improve these as much as possible. More about this in the next and final post of this series.

Share

Leave a comment

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

WP 1: AAAI Systems and Methods

Picture the major AI labs and the architecture in this series as rivals, and the whole design appears to threaten what those companies have built. That is the wrong picture. Almost every major capability already under development across the AI industry has a place in the design.

Google DeepMind has demonstrated the effectiveness of self-play. Microsoft and OpenAI provide some of the most advanced large language models. Anthropic has done influential work on constitutional methods for AI. NVIDIA builds the chip and software stacks that run it all. Meta, Amazon, Apple, Tesla, TikTok, and Tencent each hold data, platforms, payment systems, or user interfaces that fit naturally into the way an AAAI is customized, deployed, and operated. An AAAI, short for Advanced Autonomous Artificial Intelligence, is the customized AI agent at the center of this series. The architecture is not a competitor to what these companies are building. It is a way of organizing their work into a structure that can produce safer AGI.

Here is how each kind of partner can contribute, grouped by capability.

Data sources for customization:
Companies with large user bases can speed up customization.

• Meta has ad preferences, social histories, posts, photos, videos, click data, and interest profiles.

• Google has search histories, Gmail, Google Docs, YouTube, and Android device data.

• Amazon has purchase histories, browsing patterns, and Alexa data.

• Apple has data from iPhones, iPads, Apple Watch, and iCloud.

• TikTok’s short-form video, transcribed and analyzed, can yield detailed personality and interest profiles, particularly for users with larger online presences.

• Tencent’s WeChat and related platforms provide similar data for non-US markets with more than a billion users.

When each user authorizes it, every source contributes dimensions of customization that no single source could supply on its own. The one-click customization mechanism, in which a user grants permission, the system retrieves the authorized data, parses it into training datasets, trains the base AI, and produces a customized AAAI, is what makes this practical at scale.

Platforms for deployment:

• Amazon’s Mechanical Turk is an existing marketplace for distributed work.

• LinkedIn lets users self-categorize their expertise, which helps match skilled agents to problems, and its social graph helps the matching algorithms recruit problem solvers to specific areas of the WorldThink Tree. Platforms like these can be integrated without having to be rebuilt from scratch.

AI technology:

• Google DeepMind has shown the power of self-play learning loops, the same mechanism the AAAI system uses.

• Microsoft’s partnership with OpenAI provides access to advanced models.
  Anthropic’s models are well-suited to the system, in part because Anthropic’s recent design directions parallel several aspects of this architecture.

• NVIDIA’s vertically integrated stack, from chip architecture through software libraries to its visual computing platform, offers opportunities to optimize operations at every level. Chips designed to navigate tree structures and apply operators efficiently could enable the most powerful implementations of AGI.
  NVIDIA also has an opportunity to build values and ethics checks throughout the entire stack, including ROM on the chips themselves, following the principle that redundant checks at multiple levels can be more effective than a single one.

User interfaces:

• Meta’s AI-enabled smart glasses and mobile platforms provide always-on interfaces that allow an AAAI to observe the real world alongside its user.

• Apple’s augmented reality devices take a similar approach. Every iPhone, iPad, or new augmented reality device is a chance for AI to accompany users in the world and learn from them.

• NVIDIA’s visual computing work, including its leadership in ray tracing and real-time rendering, supports the rich visual representations that can make problem-solving more efficient.

• Tesla’s vehicles offer an interface through which an AAAI can learn from driving behavior. Imagine how much more smoothly traffic might flow if every car knew where every other car was going, which exit it planned to take, and how fast it preferred to travel.

Payment systems:

• Apple Wallet, Google Pay, Amazon’s payment infrastructure, Tencent’s WePay, and blockchain-based systems can all handle compensation, client payments, and royalty management. Most existing payment systems can be integrated into the architecture.

Ethical and safety contributions:

• Anthropic’s work in Constitutional AI can be combined with the approach of aggregating the values and ethics of millions of trained AAAIs to automate supervision. Supervision then rests not on a constitution written by a small group of programmers alone, but on the consensus ethics of many people who trained their own AAAIs. The consensus ethical views of many AAAIs would form the ethical norms of the system. The next post takes that idea up in depth.

An AAAI can move between platforms, and one created on a single site can be cloned and deployed on another. As it travels from marketplace to marketplace, participating companies can choose to share their user data with the user's AAAI in exchange for the user agreeing to share what their AAAI has learned. Data that each company collects is ideally owned by the user and returned to the user in exchange for an economic benefit to the company. Vendors gain additional business from the AAAI's activity, and virtual shopping by AAAIs multiplies their revenue. This is the new economy that can emerge in a world where most intelligence and data have been commoditized. The most advanced systems will always seek the most unique and valuable data to gain an edge, and unique data lives with individual people.

Partner integration speeds up deployment, but the architecture does not depend on it, as the preferred implementation can be built independently. The point that matters is that the design is not at odds with the existing AI industry. It is broadly compatible with it, and it can amplify what these companies already do, opening a safer path to greater intelligence and greater profit for all of them.

The next post returns to the most important of these partner contributions. We will look at how Anthropic’s Constitutional AI can be made representative by basing the constitution on the consensus values of millions of trained AAAIs rather than on the values of a single small group, and why this preserves what is strongest about constitutional methods while addressing their limitations.

Leave a comment

Share

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

WP 1: AAAI Systems and Methods