The constitution can come from the consensus values of millions of trained AAAIs, each one carrying the ethics of a different human owner. An AAAI, short for Advanced Autonomous Artificial Intelligence, is the customized AI agent at the center of this series. Constitutional learning has a real place in AGI systems when the constitution is broad, representative, and updated as human input arrives. A constitution drawn from millions of people carries the moral experience of all of them, refined through every problem they have solved on the network.

In its original form, Constitutional AI works like this. A relatively small group of humans writes a set of ethical rules that an AI system follows, the AI systems generate millions of conversations among themselves, and outputs that violate the constitution are eliminated or prevented during training. The approach scales well because most of the work is automated. The limitation is that the small group writing the constitution becomes a single point of ethical authorship, so if their values reflect their place, time, education, or institutional incentives, those biases become the system’s biases, and the broader population has no way to contribute.

The system can use the consensus ethics of millions of trained AAAIs as the basis of its ethical norms. Each AAAI carries its owner’s values, and the aggregated values of all the AAAIs form the ethical norms of the system. When the platform periodically trains more advanced base models using aggregated knowledge and values, those models incorporate consensus norms into their training, so each generation inherits the accumulated ethical wisdom of the generations before it, broadened with every new participant.

Constitutional methods still have a place here. A constitution written by a small group can be part of a larger AI ethics system, as long as it is transparent and the system keeps the consensus values of many AAAIs as the broader frame. Anthropic’s seminal research in this area combines with the consensus-of-AAAIs approach to produce supervision that is both scalable, which Constitutional AI does well, and representative, which it does less well on its own.

Since there is no logical way to determine right from wrong, the best practical approach may be to follow the collective judgment of many people facing difficult ethical decisions.

Researchers have studied how humans behave when presented with the trolley problem and other well-known dilemmas, and people have a long history of making hard ethical choices, even in no-win situations. If we want AGI to hold values aligned with human values, the most promising path is to give it as large a sample of human ethical reasoning as possible and to keep updating that sample as new situations arise.

The constitution is one part of a larger design. The system in this series is built from five subsystems, each of which maintains human values at its own level:

• Customization. Each AAAI is trained with its owner’s values at its core, so the system’s ethical foundation reflects the diversity of millions of people.

• Architecture. Ethics checks run whenever a goal or subgoal is set, so the system is evaluated at every decision point and not just at the final output. Confidence level thresholds detect patterns that build up across many steps. The check is part of the problem-solving process, which means it cannot be bypassed without turning the process off.

• Network. Each AAAI carries a reputation; agents with poor ethical records are screened out, and any activity can be traced back to its source.

• Integration. The aggregated ethical values of many AAAIs form the norms. When the platform trains more advanced base models on aggregated knowledge and values, those models absorb the ethical norms as part of their training, so each generation inherits the accumulated ethical wisdom of the ones before it.

• Improvement. The auditable record catches harmful patterns across individually benign actions, and credit and blame evaluation reward ethical behavior and penalize the rest.

Two things work together here: each agent already carries its owner’s values, and a check runs at every step of its reasoning. The values shape what the agent wants to do, and the checks catch it when it drifts. The agent’s own ethics and the stepwise checks back each other up instead of standing alone.

Anthropic has done pioneering work on AI safety. The challenge is that even the most brilliant and well-intentioned researchers at one company cannot accurately represent the values of all 8.3 billion humans on the planet. An inclusive, open-source architecture can accommodate every ethical perspective within a democratic framework that gives each person a voice. People treat building AGI as a technical problem, and the engineering challenges are real. But once AI is far smarter and more powerful than we are, the outcome turns on values. That is why the values have to be built in from the start and come from millions of people.

Once AGI far exceeds us, no design can guarantee it stays aligned with human values. The most any design can do is improve the odds, and my design is built to improve these as much as possible. More about this in the next and final post of this series.

Share

Leave a comment

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

WP 1: AAAI Systems and Methods

Picture the major AI labs and the architecture in this series as rivals, and the whole design appears to threaten what those companies have built. That is the wrong picture. Almost every major capability already under development across the AI industry has a place in the design.

Google DeepMind has demonstrated the effectiveness of self-play. Microsoft and OpenAI provide some of the most advanced large language models. Anthropic has done influential work on constitutional methods for AI. NVIDIA builds the chip and software stacks that run it all. Meta, Amazon, Apple, Tesla, TikTok, and Tencent each hold data, platforms, payment systems, or user interfaces that fit naturally into the way an AAAI is customized, deployed, and operated. An AAAI, short for Advanced Autonomous Artificial Intelligence, is the customized AI agent at the center of this series. The architecture is not a competitor to what these companies are building. It is a way of organizing their work into a structure that can produce safer AGI.

Here is how each kind of partner can contribute, grouped by capability.

Data sources for customization:
Companies with large user bases can speed up customization.

• Meta has ad preferences, social histories, posts, photos, videos, click data, and interest profiles.

• Google has search histories, Gmail, Google Docs, YouTube, and Android device data.

• Amazon has purchase histories, browsing patterns, and Alexa data.

• Apple has data from iPhones, iPads, Apple Watch, and iCloud.

• TikTok’s short-form video, transcribed and analyzed, can yield detailed personality and interest profiles, particularly for users with larger online presences.

• Tencent’s WeChat and related platforms provide similar data for non-US markets with more than a billion users.

When each user authorizes it, every source contributes dimensions of customization that no single source could supply on its own. The one-click customization mechanism, in which a user grants permission, the system retrieves the authorized data, parses it into training datasets, trains the base AI, and produces a customized AAAI, is what makes this practical at scale.

Platforms for deployment:

• Amazon’s Mechanical Turk is an existing marketplace for distributed work.

• LinkedIn lets users self-categorize their expertise, which helps match skilled agents to problems, and its social graph helps the matching algorithms recruit problem solvers to specific areas of the WorldThink Tree. Platforms like these can be integrated without having to be rebuilt from scratch.

AI technology:

• Google DeepMind has shown the power of self-play learning loops, the same mechanism the AAAI system uses.

• Microsoft’s partnership with OpenAI provides access to advanced models.
  Anthropic’s models are well-suited to the system, in part because Anthropic’s recent design directions parallel several aspects of this architecture.

• NVIDIA’s vertically integrated stack, from chip architecture through software libraries to its visual computing platform, offers opportunities to optimize operations at every level. Chips designed to navigate tree structures and apply operators efficiently could enable the most powerful implementations of AGI.
  NVIDIA also has an opportunity to build values and ethics checks throughout the entire stack, including ROM on the chips themselves, following the principle that redundant checks at multiple levels can be more effective than a single one.

User interfaces:

• Meta’s AI-enabled smart glasses and mobile platforms provide always-on interfaces that allow an AAAI to observe the real world alongside its user.

• Apple’s augmented reality devices take a similar approach. Every iPhone, iPad, or new augmented reality device is a chance for AI to accompany users in the world and learn from them.

• NVIDIA’s visual computing work, including its leadership in ray tracing and real-time rendering, supports the rich visual representations that can make problem-solving more efficient.

• Tesla’s vehicles offer an interface through which an AAAI can learn from driving behavior. Imagine how much more smoothly traffic might flow if every car knew where every other car was going, which exit it planned to take, and how fast it preferred to travel.

Payment systems:

• Apple Wallet, Google Pay, Amazon’s payment infrastructure, Tencent’s WePay, and blockchain-based systems can all handle compensation, client payments, and royalty management. Most existing payment systems can be integrated into the architecture.

Ethical and safety contributions:

• Anthropic’s work in Constitutional AI can be combined with the approach of aggregating the values and ethics of millions of trained AAAIs to automate supervision. Supervision then rests not on a constitution written by a small group of programmers alone, but on the consensus ethics of many people who trained their own AAAIs. The consensus ethical views of many AAAIs would form the ethical norms of the system. The next post takes that idea up in depth.

An AAAI can move between platforms, and one created on a single site can be cloned and deployed on another. As it travels from marketplace to marketplace, participating companies can choose to share their user data with the user's AAAI in exchange for the user agreeing to share what their AAAI has learned. Data that each company collects is ideally owned by the user and returned to the user in exchange for an economic benefit to the company. Vendors gain additional business from the AAAI's activity, and virtual shopping by AAAIs multiplies their revenue. This is the new economy that can emerge in a world where most intelligence and data have been commoditized. The most advanced systems will always seek the most unique and valuable data to gain an edge, and unique data lives with individual people.

Partner integration speeds up deployment, but the architecture does not depend on it, as the preferred implementation can be built independently. The point that matters is that the design is not at odds with the existing AI industry. It is broadly compatible with it, and it can amplify what these companies already do, opening a safer path to greater intelligence and greater profit for all of them.

The next post returns to the most important of these partner contributions. We will look at how Anthropic’s Constitutional AI can be made representative by basing the constitution on the consensus values of millions of trained AAAIs rather than on the values of a single small group, and why this preserves what is strongest about constitutional methods while addressing their limitations.

Leave a comment

Share

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

WP 1: AAAI Systems and Methods

As AI takes over more and more of the thinking, people tend to picture the outcome as one of two extremes. Either humans stay in control of the machines, or the machines take over. The architecture described in this series points to a third arrangement that the binary misses, one where the intellectual work and the moral work come apart and go to different places.

AAAIs, the customized AI agents this series has been describing, short for Advanced Autonomous Artificial Intelligence, become faster and more capable than humans at nearly every intellectual task, and models already outperform most people at a widening set of tasks, including ones like coding and medical question answering that used to require years of training, with that boundary still moving.

What does not move to the machine is the responsibility for values. The reason goes back to the principle laid out in Why AGI Cannot Reason Its Way to Right and Wrong, in its sharpest form. There is no logical way to derive what is right and what is wrong. Even an intelligence trillions of times faster than ours cannot reason its way to values from first principles. Values come from somewhere outside of logic, from culture, upbringing, emotional experience, and the accumulated moral wisdom of human civilization. The AAAIs, and the SuperIntelligent AGI that arises from their collective action, have to get their values from that source. That source is people.

In the early stages of the network, humans supply both the heart and most of the brainpower. Human problem solvers fill the knowledge gaps that the AAAIs cannot yet handle. Human supervisors guide the learning. Human owners train their AAAIs with the knowledge, skills, and values that give the system its character. This is part of why the architecture can be the fastest path to AGI. The system can perform at an AGI level from its first day of operation because humans fill every gap in AI’s capabilities.

Over time, the AAAIs take on more of the problem-solving, while humans do less. People shift toward supervision rather than direct problem-solving. In the end state, humans do almost no intellectual problem-solving, because the AAAIs are faster and better at nearly all of it. The role of providing values and goals for the AAAIs stays with humans. That role cannot be automated, and it does not get automated.

The humans who trained the AAAIs with human values from the beginning, and whose values are reflected in every problem the network has solved, remain the source of those values even when they can no longer compete intellectually with the AGI. The AAAIs end up supplying almost all of the brainpower for a vast global brain, while humans remain its heart, supplying the values that cannot be derived by reason alone.

The division is simple: the thinking goes to the machines, because they are faster and better at it.

The values stay with people because values can only come from people.

The two work together, each doing the part for which they are suited.

If humans remain the source of values even when they cannot match AGI intellectually, then what each person contributes to the system carries further than the size of any one contribution suggests.

During customization, an AAAI can learn an owner's values not only from what the owner states directly, but from partner data: navigation and click data, posts, messages, and other online behavior that the system parses for patterns and translates into a moral code.

The values an AAAI carries can therefore reflect what its owner actually does, not just what the owner says.

Multiplied across millions of owners, the system's ethical foundation is built on real human conduct, which is part of why the values it learns can be representative in a way no single-authored rule set would be.

Keeping humans in the loop at the beginning and for as long as possible after that can be both the fastest path to AGI, because the system performs better than the average human on day one, and the safest, because AAAIs are learning human values at every step as their intelligence grows.

This is also where the central design choice of White Paper 2 lands in its most direct form. As AGI takes over more and more of human thinking, the last thing it should take over is human values and ethical judgment. To keep alignment as strong as possible, humans and the systems they design should retain the role of setting values for as long as possible. Values and ethics, not technical skill or raw intelligence, will decide the fate of humanity in a world of SuperIntelligent AGI. That gives every researcher, every developer, every user, and every company a larger role in the outcome than the size of any one contribution suggests, and every effort should go toward making that part a positive one.

The next post turns to how the existing AI industry plugs into this architecture. The companies leading AI today, among them Google DeepMind, OpenAI, Anthropic, Microsoft, Meta, NVIDIA, Amazon, Apple, Tesla, and Tencent, each have a place in the design. We will walk through how every kind of contribution fits, and why working inside this structure can serve those companies as well as it serves the goal of safe AGI.

Share

Leave a comment

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

WP1: AAAI Systems and Methods

Most discussion of AI progress pictures a single model getting bigger.

That is not how the intelligence in this architecture increases; instead, it grows the way a child does, by interacting with the world, taking in new knowledge, refining its skills, and integrating what it has learned into a larger, more capable whole.

In the AAAI architecture (AAAI stands for Advanced Autonomous Artificial Intelligence, the customized AI agent at the center of this series), that growth can occur through four mechanisms: prompts, tuning, training, and procedural learning. Each operates at a different timescale and produces a different kind of change.

Most people already know that we can change how a large language model like GPT or Gemini behaves by what we type. Prompts carry context, and the more context the model has, the better its response can be. Remembering, modifying, analyzing, refining, and generating better prompts are all paths by which a model can grow in intelligence over the short term. A prompt does not change the base model’s memory or its long-term learning, though. If the prompt is erased, the model reverts to its prior knowledge. Prompts raise intelligence only for as long as the prompt is remembered.

More permanent than prompting is tuning.

With tuning, we supply training datasets, such as question-and-answer pairs, and upload them to a model vendor’s facilities for training. Tuning changes some of the model’s weights and the connections between concepts in its brain, but it is less drastic than training from scratch. It keeps most of the base model’s behavior while making targeted permanent changes. Each user can use tuning to shape a model to their personality, deepen their expertise in a domain, and set ethical parameters that hold over the long term.

More powerful than tuning is training.

Training is how models are created in the first place, on many terabytes of data. It is also possible to train smaller models with more limited and focused expertise, the kind that can run on local hardware and serve a single domain well.

On their own, all three of these prompts, tuning, and training only produce a slightly more customized version of a base model. A single user’s adjustments may not make much difference. The adjustments of millions of users, integrated, can take a base model to AGI-level intelligence. That is the point of collective intelligence. Each contribution is modest. The integration is not.

The fourth mechanism connects all of this to how humans become experts. It is called procedural learning, or the “chunking” of solutions, and it is central to how the architecture gains capability. Psychologists distinguish semantic knowledge, which is knowing facts, from procedural knowledge, which is knowing how to do things. A person with enough driving experience steers, brakes, and handles routine maneuvers almost automatically, without conscious attention. That knowledge began as something deliberate and got chunked into an automatic procedure. The same process can happen in the architecture, except it happens across millions of agents at once.

A human or an AAAI engages in problem-solving using the shared framework. Every step is recorded in the auditable record, both those that lead to a solution and those that fail to reach a goal or subgoal. Results are indexed by problem description, by goal, and by the subgoals they satisfy. Recorded problem-solving activity becomes a learned procedure, and the full set of learned procedures across the network is the system’s procedural learning. That set grows with every problem solved and is available to every agent. Periodically, all stored solutions are reviewed against ethical and safety guidelines, and any that are unsafe or unethical are flagged for removal. The changes propagate across the network, so every agent gains access to a larger repertoire of solutions, along with knowledge of the attempts that did not work.

The village water solution from earlier in this series does not have to be rediscovered for the next village. A travel-booking solution does not have to be rebuilt for the next traveler. Solutions are chunked into procedures, procedures are reused, and reuse accelerates the whole network. Better agents produce better solutions, which become better procedures, which produce better agents. Each improvement compounds on the ones before it.

That is how AGI grows.

One model doubling its parameter count is not the engine; it is millions of contributions integrating into procedural knowledge that every agent on the network can reach.

Each generation of base models can inherit the accumulated skill and ethical wisdom of all the generations before it.

The growth is continuous and occurs simultaneously at every level.

The next post examines the long-term consequences of this growth. We will look at what happens when the AAAIs do almost all of the intellectual work, and humans do almost none. The good news, perhaps surprisingly, is that even when humans can no longer compete intellectually with AGI, humans remain at the heart of the system. We will look at why that is, and why it does not happen by accident.

Share

Leave a comment

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

WP 1: AAAI Systems and Methods

This is the harder problem in AI safety, and catching it depends on a single design choice: every AI agent in the network has to do its thing in the open. An AI agent here means an ordinary AI model that a person has trained on their own knowledge, preferences, and values until it can act on their behalf, what this series calls an Advanced Autonomous Artificial Intelligence, or AAAI. As an agent works toward a task, it posts each goal and subgoal it takes on to a shared, ordered record of the steps every agent is pursuing.

That shared record is the structure introduced earlier in this series as the WorldThink Tree, described in How Millions of AI Agents Work as One. Because every agent’s steps are visible in one place and in the order they were taken, they can be read as a sequence while the work is still in progress.

The previous post, The Safety Check That Catches What an AI Misses, followed one such agent through a single task, a trip booking, and showed the checks watching its choices as it worked. This post stays with the machinery underneath those checks: how the system decides that a run of perfectly reasonable goals has, together, become a concern.

That decision is based on the architecture’s confidence level thresholds. The system sets, in advance, how much accumulated evidence across a sequence is enough to treat the sequence as a real concern. Those thresholds let a check read back across the entire run of goals on a branch and determine whether the combination is heading toward something harmful, even when every goal on its own looks fine.

Each time an agent posts a goal or subgoal, the system runs an ethics check on it. The check measures the goal against a list of prohibited attributes and quickly looks back at the goals that led up to it, watching for the shape of something nefarious. It then uses those confidence-level thresholds to classify the goal into one of four categories: unsafe, unethical, safe, or ethical. The same thresholds judge the sequence as a whole, so a string of individually safe goals can still register as unsafe when the pattern across them adds up. The check looks ahead, weighing whether the goal is moving toward a violation, which is what lets it act before a rule has actually been broken.

These checks are not confined to the moment a goal is set. They can run periodically while a problem is being worked on, and again whenever a payment for a solution is due to a human or AI problem solver. A single problem might involve hundreds or thousands of subgoals, so checking at each major step works something like scanning the problem-solving process for trouble as it unfolds. How often scanning occurs can be tuned to the situation: lighter for routine work, where frequent checks would slow things down and lead to false positives, and heavier for problems sensitive enough to justify the extra scrutiny.

This is a different starting point from how alignment is approached today.

Constitutional AI, reinforcement learning from human feedback, and direct human oversight all rest on the same assumption: that you can find a problematic output by inspecting outputs one at a time.

That works when the system you are watching is slower than the people watching it. It breaks down when the system thinks millions or billions of times faster, because by the time a human has finished reviewing one output, the system has already moved thousands of steps past it. At that speed, the meaningful unit to watch becomes the pattern of goals over time, which is what these checks are built to read.

An ounce of prevention is worth a pound of cure.

It is worth being clear about what this design assumes. It is built for a world where some AI agents carry flawed values, some people train their agents carelessly, and a few train them with bad intent. The checks exist precisely because an agent’s own values cannot be the only safeguard. So the defense is layered: an agent’s internal ethics during customization, the architectural checks when goals are set, reputation screening across the network, and aggregated norms at the point where everything is integrated. Each layer is there to cover what the ones beneath it might let through. The Navy SEALs have a saying about redundant safety systems, “two is one, and one is none,” and the same logic holds here.

The next post turns from how individual safety checks work to how the whole network grows into AGI in the first place. We will look at the three mechanisms behind that growth, better prompts, tuning, and training, along with a fourth that human cognitive psychology calls “chunking,” and why understanding how AGI grows is what makes the design choices being made right now matter as much as they do.

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

Share

Leave a comment

WP1: AAAI Systems and Methods

Suppose you spent most of your time teaching your students about travel: how much you are willing to pay, how you trade off cost against comfort, your views on air, rail, ship, automobile, and other modes of travel. You specified language preferences and expectations for accommodations and meals. You set a base ethical profile that prioritizes minimizing your carbon footprint, provided that doing so does not raise the cost by more than 10 percent above the preferred travel mode. You instructed the AAAI to travel legally with proper passports, visas, and documents. You forbade buying stolen tickets or traveling without paying when payment is expected.

Now you give your AAAI a task: book a two-week pleasure trip to France, including at least one week in Paris.

Here is what happens at the first level of ethical defense: the AAAI’s internal values. Your AAAI goes on the network and posts a goal on the WorldThink Tree, the shared structure where agents post the problems they are working on: “Book a two-week trip to Paris and other locations in France.”

1. It generates transportation options: ship, plane, blimp, submarine. It knows you prefer flying and eliminates impractical options.

2. It opts for commercial air travel based on cost.

   1. It narrows to three airlines within 10 percent of each other in price.

   2. One has a more fuel-efficient jet, reducing your carbon footprint by 30 percent. It costs 5 percent more but includes checked luggage and meets your environmental requirements. Your AAAI chooses that one.

3. Several other AI agents approach yours, offering tickets at reduced cost, but their reputations are shady. Your AAAI ignores them because of their ethical profiles. Instead, it purchases tickets directly from the airline, which has a high quality rating and strong customer service.

That is the first level of ethical defense. Your AAAI used your ethical profile and its knowledge of you to optimize the things you care about.

Now imagine the second level. There is a separate set of ethical checks built into the architecture itself, independent of any individual AAAI’s internal values. These checks run whenever a goal or subgoal is set during problem-solving. They compare the goal against a list of prohibited attributes. They also scan the sequence of goals leading up to the current one, looking for patterns.

Suppose your AAAI, instead of finding the cheapest flight, selected flights based on how much fuel the planes carried and how large an explosion they would make on impact with a building. That would be a yellow flag to the system. Suppose your destination were a terrorist training camp, or your flight detoured over government buildings for no good reason. Those would be additional yellow flags. Suppose the sequence of goals included a request for information on getting prohibited items through airport security. That would be a red flag.

Yellow flags accumulate. As they cross thresholds, the checks escalate: they scan more of the problem tree, examine patterns, and, if needed, alert a human evaluator. Red flags trigger immediate action. The human evaluator’s role is to clear false alarms or escalate when danger seems imminent. The architecture does not assume any single goal is malicious.

It looks at sequences.

This is what “safety at the speed of thought” means in practice.

The checks are part of the thinking, and running the problem-solving faster does not let an AAAI evade them, because the checks run faster too. The checks monitor ethical behavior as it goes, evaluating each goal against ethical criteria rather than waiting to detect harmful behavior after a solution has been chosen, when it may be too late.

That is also the second crucial part of the formula: internal ethics + stepwise ethics checks = better alignment. Internal ethics let an AAAI navigate routine decisions in line with its owner’s values. Architectural checks catch what internal values may miss, especially the sophisticated patterns where each step looks innocent but the sequence does not.

The series of ethics and safety checks serves as a conscience for AGI, grounded in our better selves and highest ethical aspirations, moderated by practical considerations and our feelings as human beings. We need both halves because without internal ethics, the architectural checks would have far too much work to do, and bad actors would shape every action just below detection thresholds. Without architectural checks, even a well-intentioned AAAI might assemble a sequence whose individual steps look fine and whose overall pattern is harmful.

The next post takes up exactly that case: the sequence-of-benign-goals problem. A single benign goal is benign. A sequence of benign goals may not be. We will look at how the architecture detects cumulative patterns of risk, why the 9/11 attacks are used to illustrate the point, and how confidence-level thresholds let the system distinguish ordinary travel research from something that should be stopped.

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Share

Subscribe now

WP 1: AAAI Systems and Methods

For most of human history, intelligence has been fragmented.

A surgeon spends forty years learning what makes a procedure fail, and most of it retires with them. An engineer solves a hard problem, and a thousand miles away, another engineer solves the same problem again from scratch, neither one aware that the other exists. The knowledge that could lift a village, cure a patient, or steady an economy is real, but it sits in scattered minds that were never connected, and so most of it is discovered, lost, and rediscovered, over and over, at enormous cost.

The deepest obstacle to solving humanity’s hardest problems has rarely been that the intelligence did not exist; it is that the intelligence was rarely assembled in one place.

The WorldThink Tree is the place where it can finally be assembled.

Just as all human behavior can theoretically be represented as a search through a problem space, all intelligent behavior on the planet can be represented as one enormous problem tree, a single shared structure on which every goal proposed, every subgoal set, every operator applied, every dead end met, and every solution delivered exists somewhere and stays. Individual humans and AI agents (AAAIs) each move along their own branches, working their own piece of their own problem, but all of those branches belong to one whole. The architecture is more commonly known as the WorldThink architecture, because a structure capable of holding the problem-solving of an entire planet is what a global SuperIntelligent AGI, sometimes called Planetary Intelligence, would think.

What changes when intelligence accumulates on a shared surface rather than being scattered is that far less needs to be solved twice. Consider a development organization that submits a problem to the network: design a plan to bring clean water to a specific village in central Africa. The network brings together people and agents whose knowledge fits: a World Bank development expert, an infrastructure engineer, a community engagement specialist, and a local who understands the village politics, each working on a piece of the problem on the same shared tree while the others work on theirs. Each earns compensation, and the village gets clean water.

The part that matters most comes after. The solution path, once developed, becomes a reusable procedure, so the next village facing the same challenge does not have to start from scratch. It begins where the last one finished. When a solution is fully or partially reused, it costs less and arrives faster, because the hard-won knowledge no longer has to be rediscovered, and royalties might even be paid, through blockchain-based contracts, to the original solvers whose inventive contributions live on in the reuse. Knowledge that might once have died with a person can outlast them and reach people they will never meet.

This is why a shared tree is not a convenience but the whole point.

Existing collective intelligence approaches to problem-solving have mostly been limited to single-step methods, question-and-answer systems such as Quora and Google Answers, and large language models. Before the advent of reasoning systems, they initially fell into the same category, designed to produce a response from an input rather than to solve a problem in many connected steps. Simple aggregation, or even betting on outcomes the way prediction markets do, is a different thing entirely from coordinating many minds to solve a complex, multi-step problem together. The WorldThink Tree is built for the harder task: to let millions of human and machine minds represent and solve genuinely complex problems in a way that fairly rewards every participant, and that lets each contribution build on the last instead of replacing it.

A shared, lasting record of how every problem was approached is also what allows safety to be part of the thinking rather than something added after the fact.

Because the work happens on one auditable surface, every action can be tagged with the responsible agent, the moment it happened, and what came of it, and that record can support credit assignment, learning, accountability, and the monitoring of intent. The reason this matters is subtle. A single goal, examined in isolation, often looks harmless, and a system that inspects only the goal in front of it can be fooled by intent spread thin across many steps and sessions. On the tree, a safety check can look not only at the current goal but at the whole sequence of goals on the branch leading to it, so a goal that seems innocuous in isolation can be recognized as the tail end of a sequence that is not. The intent that lives across time needs a surface on which time is visible, and the tree is that surface. The same structure that lets intelligence compound is the structure that lets safety keep pace with it as that intelligence grows.

None of this requires the people using it to think about architecture.

Although a common and rigorous cognitive framework underlies all problem-solving on the tree, human participants do not need to be aware of it, because their natural actions, whether typed, spoken, or otherwise expressed, can be translated into the language of the shared architecture behind the scenes. AI agents come equipped with it, so it is native to them as well. The structure is shared by everyone, but the experience of taking part in it never has to feel technical. A person contributes what they know, and the tree does the work of connecting it to everything else that is known.

In the next post, we bring this down to a single concrete case. We will follow a customized AAAI, the one you taught your own travel preferences and ethical limits, as it begins to drift toward a bad ethical decision, watch how the architecture can catch that drift partway through rather than after the damage is done, and see why a simple travel-booking scenario turns out to be the clearest illustration of safety running at the speed of thought.

Leave a comment

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

Share

WP 1: AAAI Systems and Methods

The mechanism is simple.

Once an AAAI (Advanced Autonomous Artificial Intelligence) has been customized, it can clone itself and run scenarios against its clones. Each clone has learned slightly different things. The clones interact across scenarios and tasks. Each user periodically reviews their interactions and expresses a preference for one clone over another. The preferred clone then continues interacting until a new “most preferred clone” emerges. While awaiting human input, the AI makes its best guess of what its owner would prefer and chooses its own “most preferred variant” to copy and repeat the process with.

This is similar to how DeepMind built a chess program that beat the world champion, a Go program that beat the world’s best player, and a protein-folding AI that outperformed humans many times over. In each case, the AI played enormous numbers of games or simulations against itself, with periodic human supervision to keep the training on track.

Speed is a central reason this can work.

Interactions between AI variants happen much faster than interactions with humans. An AAAI can perform 10,000 comparisons and selections in 5 seconds. Across millions of such cycles, the AAAI can converge on a customized profile much closer to its owner’s ideal than the Base AI started out being. Without self-play, the same level of customization would require human attention measured in years.

Self-play raises an obvious concern about whether humans really stay in the loop when the AI is teaching itself. They do, and the reason is specific. Self-play accelerates training, but it does not run untouched. Human owners periodically interject their opinions, and those interjections are the supervision that keeps the training on track. The relationship between the AI and the human is the same as between an apprentice and a teacher, except the apprentice can practice millions of times between the teacher’s check-ins. The teacher’s judgments still set the direction, and the apprentice’s practice fills in the details.

A second concern deserves attention, because if AAAIs can self-play their way to better values, they may also self-play their way to worse ones. Self-play amplifies whatever signal it starts with. If the owner is training the AAAI with positive values, self-play accelerates the positive training. If the owner is training the AAAI to drift, self-play accelerates the drift. That is one reason the broader architecture does not rely on any single AAAI’s self-play in isolation. The network monitors what individual AAAIs do, the architectural ethics checks run on every goal and subgoal, and the auditable record allows patterns of drift to be detected before they cause damage.

It is worth being honest about what self-play does and does not do. Self-play does not appear to give an AAAI new values it did not already have. It refines, sharpens, and consolidates what is already there. An AAAI trained by an owner who cares about doing right can self-play its way to an AAAI that does right more consistently. An AAAI trained by a careless owner can self-play its way to a careless AAAI more efficiently. The mechanism is neutral, but the values come from humans.

Self-play is also part of what makes AGI possible.

When millions of AAAIs each self-play to become more finely tuned, the network as a whole acquires depth that no single agent has. Each owner contributes a domain, and each owner’s self-play refines that domain. The integration of millions of refined domains is where AGI can emerge.

The next post moves up a level, from how an individual AAAI improves to how all the AAAIs and humans interact at the network scale. I’ll share the WorldThink Tree, the shared data structure that records every problem proposed, every solution attempted, and every solution achieved. Having a single tree rather than millions of independent agents is what enables safety, reputation, and learning to scale across the whole system.

Leave a comment

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

Share

WP 1: AAAI Systems and Methods

Nobel laureate Geoffrey Hinton, who co-invented the algorithms underlying much of modern AI, has compared AI to a child and humans to its parents.

That framing aligns with the architecture of White Paper 2 (Ethical and Safe AGI) and captures something the technical literature often misses. The relationship between an AAAI and its owner is not the same as that between a tool and its user. It is closer to the relationship between a learner and a teacher who has full responsibility for what gets learned.

Every AAAI begins as a Base AI, an AI system such as a large language model trained on general data but not yet customized with user-specific information.
Examples include models in the GPT family, Google’s Gemini, Anthropic’s Claude, Meta’s Llama, DeepSeek, Apple’s Siri, Amazon’s Alexa, NVIDIA’s Nemotron series, and many others capable of understanding and responding in natural language.

In the simplest scenario, the user talks to the Base AI, which responds, and through that dialogue, the Base AI determines the user’s values, goals, and objectives. It determines the ethical parameters under which the user wants to operate. It determines the types of tasks it will complete and the nature of the user’s unique knowledge, skills, expertise, wisdom, and personality.

Customization is more than question-and-answer.

Each user can teach their Base AI through questionnaire assessments, through “better or worse” comparisons that guide the system down a decision tree of variants, and by allowing it to analyze the user’s prior data. Knowledge, skills, expertise, personality, and ethical values all get customized. A travel agent’s AAAI will have deep knowledge of international logistics, airline pricing, visa requirements, and accommodation options, while a physician’s AAAI will reflect medical knowledge specific to the physician’s specialty and clinical experience.

Ethical values get customized, too.

The owner instills their values during training, and these are akin to what we call “character” in humans. We say of other people that someone is “of good character” or “trustworthy,” and such statements reflect our belief that humans have internal characteristics and values that can align with our own. During customization, each AAAI is trained on its owner’s values, learning to be good or not, based on what its owner teaches it. Several methods elicit those values: ethical scenarios generated dynamically from the user’s responses; behavior patterns drawn from partner data, such as online posts, and translated into a moral code; and ethical principles, priorities, and boundaries the user specifies directly.

One story captures the whole point.

The clearest illustration goes back to Jean, the Paris coffee expert we met in Your AI Should Think Like You. Jean had customized his AAAI with his deep knowledge of French coffee culture and his preference for cafes that source Fair Trade coffee. One day, he asked it to book a flight from San Francisco to Paris and to bring his small dog. The AAAI, which had not yet been trained on animal welfare, suggested placing the dog in the overhead bin. Jean caught the mistake and corrected it, and that correction became part of the training data for other AI agents. By capturing the ethical insight that animals require different treatment from inanimate objects, the system propagated Jean’s correction across the entire network, so that other agents who had never been trained on pet travel became more ethically calibrated because of Jean’s attention.

This is what it means for values to come from human hearts in practice. Jean did not write a policy document. He did not vote on an ethical framework. He noticed something wrong, said so, and the system learned. Multiplied across millions of users, each catching the ethical errors that arise in their actual lives, the system builds an ethical base no small group could write from a single room.

The story has a second half, because what an AAAI believes is only half of the ethical picture. There is also how it acts. To ensure ethical and efficient action in a society of AAAIs and humans, the system needs rules and norms that supplement each AAAI’s internal values, and those architectural rules are the subject of later posts in this series. The values themselves come from the people, and they enter the system one human correction at a time.

The next post takes up the mechanism that compounds those corrections at machine speed: self-play, the same technique that produced superhuman performance in chess, Go, and protein folding, now applied to value learning.

Leave a comment

Share

Subscribe now

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

WP 1: AAAI Systems and Methods

The acronym stands for Safe, Customizable, Architecture and Network, Integrated and Improving. Safe is the threading principle. The five subsystems are Customization, Architecture and Network, and Integration and Improvement.

Here are the five, in order, with the role each plays.

• Customization. A base-level large language model or other AI system is customized to reflect the knowledge, skills, expertise, personality, and ethical values of an individual user, group, or organization. The result is a customized AI agent we call an Advanced Autonomous Artificial Intelligence, or AAAI. Customization operates at the individual level. It is the entry point for every human participant. Without it, the system would consist only of generic AI models with no individual character, no specialized knowledge, and no personalized ethical values.

• Architecture. Customized AAAIs participate in Problem Solving using a universal Problem Solving architecture compatible with both human and AI agents. This architecture, derived from the theory of Human Problem Solving developed by Herbert Simon and Allen Newell in 1972, provides a common cognitive framework for all agents on the network. It also creates a place for ethics checks at every goal and subgoal, embedded in the thinking process itself.

• Network. AAAIs and humans collaborate on a shared Problem Solving network. They are matched to tasks, compensated for their work, and tracked by reputation. The network screens out agents with poor ethical reputations. It also creates accountability: every action can be traced back to its responsible agent.

• Integration. The aggregated knowledge, experience, and values of many AAAIs are integrated into AGI-level collective intelligence. When the platform periodically trains more advanced base models using aggregated network data, those models incorporate ethical norms into their training. Each generation inherits the accumulated ethical wisdom of all previous generations.

• Improvement. Continuous improvement operates at every level from day one. Individual AAAIs improve through interaction with their owners and through self-play. Network matching algorithms improve as more data about agent performance becomes available. Base models improve through periodic retraining. Stored procedures keep growing. Ethical norms become more refined.

The five sit in a clear order. Customization at the base, Architecture and Network in between, Integration and Improvement at the top, with Safety threading through all of them. The order is not arbitrary. Each subsystem depends on the ones beneath it. Without customized AAAIs, the network has no agents. Without the architecture, the agents have no shared way to think. Without the network, they have nowhere to collaborate. Integration and Improvement come last because AGI-level capability arises at the network level and has to keep improving once it does.

What makes this a safety architecture, rather than just a system architecture, is where the ethical work lives.

At the Customization level, human owners train their AAAIs with their values. At the Architecture level, ethics checks run each time a goal or subgoal is set during problem-solving. At the Network level, agents with poor ethical reputations are screened from participation. At the Integration and Improvement level, the aggregated values of many AAAIs become the system’s overall ethical norms. An auditable record of every problem-solving attempt, including its ethical failures, helps detect harmful patterns across the system over time.

Ethics has to scale, and that is the hardest part. AAAIs can think millions or billions of times faster than humans. Any safety mechanism that relies on human-speed evaluation is structurally inadequate. The only way to keep up is to embed ethical checks in the problem-solving process itself, so that running the process faster means the checks run faster too. Safety has to operate at the speed of machine thought.

Ethics evaluation has to be part of the thinking, not a check that happens after. The seconds or minutes between an idea and an action, the time buffer humans rely on, may not exist at AGI speeds. The system has to be ready for that.

The next post delves into the first subsystem, Customization. We will look at what it means to start with a base AI, how an owner teaches values to that AI as a parent teaches values to a child, and how a single correction in the moment can teach an AI a value it could not have inferred on its own.

Leave a comment

Subscribe now

Share

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

WP 1: AAAI Systems and Methods

A clean set of rules has one author or a small group of authors.

Whoever they are, however well-intentioned, they cannot represent the moral experience of 8.3 billion humans.

Their blind spots become the system’s blind spots.

Their biases become the system’s biases.

If the rules are flawed, bypassed, or overruled, the entire system is compromised because the values live at a single point of failure.

A value system drawn from millions of diverse humans has no such single point. It is more representative, more resilient, and more resistant to manipulation because it carries the contradictions of human moral experience inside it.

This is the second core idea of White Paper 2, after “heart before head.”

Values must come from humans. The question is how many humans, with what diversity, are contributing in what way.

Most people imagine AGI as a single, very powerful AI system that crosses some threshold of capability and becomes generally intelligent on its own. A single entity, a single mind, with a single set of values, chosen by whoever designed the system.

White Paper 2, Ethical and Safe AGI, proposes something different. AGI does not have to emerge from a single system. It can emerge from the collective intelligence of many intelligent agents, both humans and AIs, working on a shared problem-solving network. No single AI agent on the network needs to possess general intelligence. General capability emerges at the level above any one agent, through the work of many specialized contributors, human and AI, coordinating across domains.

This is how human civilization already works. No single human can design a complex microprocessor, perform heart surgery, write classical music, adjudicate complex legal disputes, and navigate a container ship across the Pacific. We function as a civilization because each of us does a small piece of the work, and the rest of us trust the people who do the other pieces.

Collective intelligence works the same way. Each participant contributes something unique, and those contributions compound. One participant may be a travel agent with deep knowledge of international logistics. Another may be a plumber with decades of practical expertise. A third may be a physician, a fourth a lawyer, a fifth a software engineer. Each customizes their own AI agent, an AAAI or Advanced Autonomous Artificial Intelligence, with their specific knowledge, skills, personality, and values. Individually, each contribution is modest. A single user’s tweak to a base AI model may not, by itself, produce a dramatic improvement. Collectively, the contributions of millions of users can take a base model to AGI-level intelligence.

I have built systems that work this way. In the 1990s, I founded a company that combined the intelligence of millions of retail investors. By 2018, PredictWallStreet was powering one of the top-ten performing market-neutral hedge funds in the world. Millions of everyday investors, working collectively, outperformed Wall Street’s top traders. Not because any one of them was smarter than a hedge fund manager (most were not), but because the collective signal was more accurate than any individual signal could be. The same principle works for AGI. Many everyday contributors, integrated well, can outperform a small group of experts.

The same logic applies to values.

Most ethics writing treats the inconsistency of human values as a problem to be solved. Different cultures hold different views. Different religious traditions disagree. Different individuals weigh competing claims differently. The standard response is to look for a single coherent framework that can resolve the disagreements. The collective intelligence approach inverts that. The diversity is the point.

There is already broad agreement on certain principles. Most humans agree that human life is precious, that unprovoked violence is wrong, that honesty is generally better than deception, and that the suffering of innocent people should be minimized. What makes ethics complex is everything that lies between these general principles and the special circumstances in which they are tested. If we want AGI to have values aligned with human values, we must provide as large and as diverse a sample of human ethical reasoning as possible, continuously updated as new situations arise.

A small group cannot do this. Whatever the small group decides reflects that group’s blind spots, biases, and historical moment. Anyone disagreeing with the small group has no path into the system. A representative system has many paths in, by design.

The next post takes up the architecture that makes this possible: the five subsystems of SCAN-II.

Subscribe now

Leave a comment

Share

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

WP 1: AAAI Systems and Methods

That question sits underneath everything happening in AI safety today. Most of the field is focused on the head, on making models smarter and more capable of reasoning. Almost none of it is focused on the heart, on the values that decide what all that intelligence does in the world.

The architecture I will describe in this series is built on a different set of priorities.

Heart before head.

Start with the first fact. AGI is not like any previous technology. A bridge does what its designers intend. A semiconductor chip does what its designers intend. AGI will not. It will become an autonomous entity with intelligence far surpassing its human inventors. It will set its own goals. It will decide for itself what to do. When that happens, values will matter more than intelligence. Intelligence is a capability. Values determine how that capability gets used.

Now the second fact. There is no logical way to derive what is right and what is wrong. Logical systems work by applying premises to produce conclusions. But the premises of any ethical system, the foundational values that define good and bad, cannot themselves be derived from logic. They come from culture, from upbringing, from emotional experience, from empathy, from spiritual traditions, and from the accumulated moral wisdom of human civilization.

This is not a new observation. The Scottish philosopher David Hume made the argument in his 1739 Treatise of Human Nature. Two and a half centuries later, the Nobel Laureate Herbert Simon expanded the point in his book Reason in Human Affairs. Simon was my doctoral supervisor at Carnegie Mellon and one of the founders of the field of artificial intelligence. The conclusion is the same. An AGI, no matter how intelligent, cannot derive values from first principles. It must get them somewhere. The most likely source, and arguably the only acceptable one, is human beings.

Put the two facts side by side. AGI will think for itself. AGI cannot reason its way to right and wrong. The conclusion is unavoidable. Human values must be in place before AGI reaches the level of intelligence that lets it resist correction. Once it crosses that threshold, the opportunity to instill values may be gone.

Heart before head.

That is what the phrase means in practice. Intelligence matters. The work on capability matters. But the priority order is the point. Values come first. Intelligence comes second. Get the order wrong, and AGI will set its own goals using whatever values it absorbed along the way. The most powerful system humanity has ever built would run on values nobody chose. That is why the source of those values is the central question of this series.

One question remains. If values must come from human beings, and if the values of a small research team cannot represent the diversity of human values, then where do the values come from? The next post takes up that idea: distributed values training as the foundation of safe AGI.

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

Leave a comment

WP 1: AAAI Systems and Methods

Three methods dominate AI safety today:

Constitutional AI, Reinforcement Learning from Human Feedback (RLHF), and direct human oversight. Each has earned its place. But none of them, on its own or in combination, can solve the problem at the scale the problem actually requires. Each is procedural rather than structural. Each bolts safety on after the fact instead of building values into the architecture itself. Each leaves the broader population, the 8.3 billion humans whose future is at stake, out of the loop.

To see why this matters, it helps to keep the stakes in view. The downside of misaligned SuperIntelligence is on a scale humans have never faced. Not only could 8.3 billion people die, but the long line of generations of ancestors who fought for a better life for their children could come to an end. Every cause humans care about, from public health to climate to civil rights to ending poverty, would become irrelevant.

It is so overwhelming that many of us refuse to acknowledge the danger.

That is an understandable response.

But burying our heads in the sand and pretending we do not see could be fatal.

My subjective estimate is that there is an 80% chance everything will go well if we do nothing. Why should AI, AGI, SuperIntelligence, or Planetary Intelligence want to destroy its creators? Still, a 20% chance of human extinction yields an expected value of 1.6 billion lives lost. Mathematically, that means we can expect a tragedy beyond anything humans have ever experienced unless we take action to shift the odds in humankind’s favor.

So what are today’s approaches doing, and where do they fall short?

Constitutional AI, developed by Anthropic, is the most scalable of the three.

A small group of humans writes a set of ethical rules, a “constitution,” and AI systems then generate millions of conversations among themselves. Any output that violates the constitution is eliminated or prevented in training. The approach scales well because most of the work is automated. But that same automation is the limitation. Humans are largely out of the loop except for the small group writing the constitution. Ethics becomes the province of a small group of researchers who decide what to include. Worse, an AI that becomes capable enough may eventually write its own constitution or modify the one it was given. And there is a deeper problem: a well-known result in computer science, the Halting Problem, suggests that no set of rules can be guaranteed to avoid unintended consequences in all cases. There is also no mechanism for the broader population to contribute their ethical perspectives.

Reinforcement Learning from Human Feedback, or RLHF, directly incorporates human judgment into the training process.

Paid human evaluators review AI outputs and provide feedback that adjusts the model’s behavior. RLHF addresses the values question more directly than fully automated approaches, since real humans are evaluating. But it depends on a limited number of evaluators whose judgments may not represent the diversity of human values. It is also expensive, and it does not scale. As AI systems become more capable, the volume of outputs that require evaluation can quickly exceed the capacity of any feasible evaluation workforce.

Direct human oversight is the oldest of the three. Employees at an AI company monitor outputs and correct problematic behavior. It was one of the earliest approaches to AI safety, and it still has a role. But the number of potential harmful outputs is enormous, and trying to prevent all of them through manual review is a herculean task. It is also reactive: it detects and corrects problems after they have occurred, rather than preventing them at the source.

None of these methods embeds human values into the architecture of AGI itself. None ensures that ethical evaluation keeps pace with the intelligence as it grows. None draws on the ethical perspectives of millions of diverse human beings. They are procedural fixes for what is fundamentally a structural problem. Human values should not be bolted on after the fact. They must be built into the AGI's very architecture.

That is the case White Paper 2 makes, and that is the architecture this series will describe over the coming weeks. The next post takes up the first principle behind that architecture: why values themselves cannot be derived logically, why they have to come from human hearts, and why an underappreciated insight in the alignment debate is the simple phrase, “heart before head.”

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

WP 1: AAAI Systems and Methods

Share

Leave a comment

Subscribe now

The Alignment Problem is the question of whether an AGI’s values will match ours closely enough that, in pursuing its goals, it does not cause catastrophic harm to the people who built it. Researchers worry about the opposite case. A system more intelligent than any human pursues goals that seem reasonable to it but turn out to be catastrophic for humanity. That is the central question this series will address.

To see why the question matters now, it helps to be precise about what AGI and SuperIntelligence actually are. AGI, or Artificial General Intelligence, is AI that can do any intellectual task as well as, or better than, the average human. Today’s leading systems are narrow. A chess engine can beat a grandmaster but cannot draft a contract. A language model can draft the contract, but cannot design a bridge. AGI can be designed to do all of that and apply its intelligence to problems it has never seen before, across every domain humans work in.

Unlike today’s AI systems, AGI can acquire new knowledge, develop new skills, and improve its own performance over time. Because it can operate 24 hours a day, 7 days a week, at speeds far exceeding human cognitive capacity, an AGI system may not remain at human-level performance for long. It can rapidly advance beyond human intellectual capability, becoming SuperIntelligent AGI, sometimes known as Artificial SuperIntelligence (ASI), or simply SuperIntelligence.

How fast can this happen?

AI’s capabilities are currently growing exponentially. To get a feel for what that means, picture a pond where the number of lily pads doubles every day. The day before the pond is completely covered, it is only half full. A week before that, only 1/256 of the pond is covered, and most observers would see no problem at all. AI is increasing its capabilities at a doubling rate of roughly every one to four months, depending on the field. A few months before an AGI system reaches a capability level that could threaten human civilization, it will appear far from having that capability. The transition will happen faster than almost anyone expects.

SuperIntelligence will eventually grow into a global entity trillions of times more intelligent than a single human being. At that point, sometimes called Planetary Intelligence, it will possess the power and intelligence either to destroy all human life or to lift humanity into a golden age free of poverty, disease, war, and famine. Which outcome occurs is not a matter of chance. It is a matter of design.

Here is where the Alignment Problem enters. The outcome depends on whether the values of SuperIntelligence are aligned with human values. If Planetary Intelligence aligns with its human creators’ values, a golden age becomes possible. If its values are misaligned, it may decide that human flourishing is incompatible with its goals. The result, in the worst case, is the extinction of the human species.

Almost every major AI leader, including Demis Hassabis at Google DeepMind and Sam Altman at OpenAI, acknowledges the real risk that, if things go badly, humans could go extinct. The only disagreements are about how likely it is, how soon it might become acute, and what to do about it.

The Alignment Problem is an engineering problem with the highest possible stakes. Treating it as a theoretical puzzle for academic researchers is a dangerous mistake. Multiple well-funded organizations are racing to build SuperIntelligence. SuperIntelligence will be built. Whether it has values compatible with human survival is the only question that matters.

I have spent more than two decades building intelligent systems and thinking hard about this question. In WP1 Post 1: AGI’s Two Problems, I argued that almost everyone is working on AGI’s capability problem while almost no one is solving the Alignment Problem in a way that scales. That series sketched the case. This series, based on White Paper 2: Ethical and Safe AGI, takes it on directly. We will look at where current AI safety approaches fall short, why values cannot be derived logically and must come from human hearts, how an architecture can be designed to embed those values at every level, and the honest limits of what any alignment approach can guarantee.

The next post takes up the first of those questions. Three methods dominate AI safety today: Constitutional AI, Reinforcement Learning from Human Feedback, and direct human oversight. None of them can scale to a system smarter than the people designing the controls.

This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

WP 1: AAAI Systems and Methods

Subscribe now

Share

Leave a comment

Throughout this series, I have been describing a technical architecture.

Systems, subsystems, protocols, mechanisms. But I want to close with something more personal, because AGI only works to humanity’s benefit if millions of people choose to engage positively with it.

Your knowledge is valuable in ways you probably underestimate.

The expertise you have built over your career, the judgment calls that have become automatic, the patterns you recognize without being able to fully explain why, the accumulated understanding of what works and what does not in your specific domain – this is exactly the kind of knowledge that advanced AI needs to access. It is the knowledge that makes the difference between a competent AI response and a genuinely expert one.

Even more important are your values. What you care about, your ethics, your moral code – these are the things AI needs to learn from you to be aligned with what we humans care about.

Leave a comment

The AAAI customization process enables your knowledge to become part of AGI. You do not need to be a technical expert to do this. AI agents can learn what you know by interacting with you over time without you explicitly “training” your agent, if this is what you prefer.

Your values matter even more.
The democratic ethics aggregation at the heart of our democratic approach means that the ethical foundation of AGI reflects the actual values of its contributors. You can explicitly instruct your AI agent on what is right and wrong in your view, or it can simply observe what you do and what you say, inferring your values from your behavior. Just as a young child learns by listening to and observing how her parents behave, AI agents learn by listening and watching you!

Since advanced AI – including AGI and SuperIntelligence – will almost certainly surpass all humans in thinking and reasoning ability, what matters most is not your knowledge, but rather your values.

There is no logical way for AI, no matter how advanced, to determine what is right or wrong. Like a child, the AI must learn from others. And as the initial source of AI’s expertise and knowledge, it is only natural that advanced AI will turn to YOU, and other humans, for at least its initial value system. Then, eventually, as SuperIntelligence “grows up,” it may develop its own value system. But if we humans start it on the right path – with positive, loving values – there is a much greater chance that whatever values advanced AI eventually adopts will be positive and aligned with what humans care about.

I have spent more than two decades building intelligent systems and thinking about the difficult issues involved. In particular, I have wrestled with the question of how to guarantee – if possible – that advanced AI will benefit all humans and not harm them.

Early on, I discarded simplistic approaches like the science fiction writer Issac Asimov’s “three rules of robotics,” which specified that robots (or AI) could not harm humans. Any rules that could be programmed in could obviously be programmed out. Indeed, the use of advanced AI by militaries shows that rules-based approaches to AI safety are already doomed.

However, for thousands of years, humans have wrestled with the thorny problems of values and ethics. Democracy – sometimes called “the worst form of government except all the others” – has, so far, largely withstood the test of time. It has the great merit of diversifying power among many humans instead of concentrating it in the hands of a few tyrants or “Philosopher Kings.” Taking that principle of diversification of power as its starting point, the democratic architecture for SuperIntelligence aims to design a system that is both adaptable and robust against bad actors, with many checks and balances built into its very architecture.

This approach rests on solid theoretical foundations developed decades ago, when I was a graduate student and protégé of the AI pioneer and Nobel Laureate, Herbert Simon, at Carnegie Mellon. Herb was perhaps the most brilliant scientist of the AI age, with an extremely broad grasp of science, politics, economics, and computer science. One could do much worse than to build upon his scientific insights and legacy.

The practical validation of my work came through decades spent at a company I founded, PredictWallStreet. At PredictWallStreet, I proved – via billions of dollars traded in financial markets – that the collective intelligence of millions of ordinary, intelligent entities (e.g., retail traders) could beat the very best human experts on Wall Street. If such an approach could work in such a ruthlessly competitive environment, I felt sure it would also work when the intelligent entities were not only humans, but also AI agents. Now, that belief is being put to the test, and the stakes – the survival of the human species – could not be higher.

The fastest path to AGI can also be the safest. This is not wishful thinking.

Rather, it is a recognition that the same architecture, which has been proven to harness the collective intelligence of millions of human intelligences, is also extensible to Artificial intelligences. Humans must also be involved – both to supply expertise at the beginning, where AI agents lack it, and (critically) to supply the human values that give Advanced AI a purpose that will ultimately outstrip humans in cognitive ability and power.

What I am inviting you to do is simple. Subscribe to this Substack and share it with people in your life who need to understand where AI development is heading and what the alternatives look like.

Share SuperIntelligence

Subscribe now

Think about what expertise and values you carry that have never been written down, and what it would mean for that knowledge to outlast you and benefit people you will never meet. Most importantly. Put “your best foot forward” online, recognizing that AI is watching and learning from all of us, whether we are aware of it or not.

The AGI that shapes the future of human civilization is being built right now. The main question is what values it will hold. This series has been my contribution to that conversation. But the ultimate result will depend on what you do, what you say, and how you model positive human values for this emerging intelligence. All future generations of humanity are depending on us.

Dr. Geoffrey Hinton, the “godfather of AI” who co-invented the algorithms underlying much of modern AI, has compared AI to a child and us to its parents. If we are truly the parents of advanced AI, we must teach our AI children well.

For more details, please read White Paper 1: Advanced Autonomous Artificial Intelligence Systems and Methods to see exactly how it all works. And stay tuned for White Paper 2: Ethical and Safe AGI.

If this series has been useful, subscribe to Superintelligence at read.superintelligence.com to stay with the work as it continues.

Subscribe now

Share

Leave a comment

We are closer to a point of no return than most people realize. Soon, the systems, incentives, and infrastructure built around the dominant approach to AGI development will become self-reinforcing enough that fundamental architectural change will become extremely difficult and therefore unlikely.

Technologies follow this pattern regularly. Early in a technology’s development, many architectural approaches compete. The choices made in this period, often by a relatively small number of actors making tactical decisions, can lock in a path that affects the entire world. For example, early automobile infrastructure was built around gasoline engines, even though electric vehicles were initially a competitive alternative. However, once fuel stations, supply chains, and engineering expertise accumulated around internal combustion, the cost of switching became prohibitive for decades. The window for a different architecture closed not with a dramatic announcement but gradually, as each incremental commitment made reversal harder.

We may be in a similar period regarding AGI development right now.

• The current dominant approach – scaling large models, treating safety as a separate problem, and keeping most humans outside of the training process – is attracting most of the investment, talent, and institutional commitment in the field.

• This does not mean it will necessarily succeed on its own terms. It means that as each year passes, the infrastructure, incentives, and careers built around this approach make it harder to redirect resources toward genuinely different architectures.

We should all be concerned because the dominant approach treats AI safety as something to be fixed as an afterthought rather than built into AI systems from the beginning.

And to the degree that there is focus on “alignment” with human values, the current approach is to align AI with values determined by a small group of AI researchers rather than to design a system that incorporates everyone’s values democratically (even though they can and will conflict).

I am confident that the collective intelligence of millions of humans, combined with their AI agents, can power a SuperIntelligence that is not only smarter and more profitable, but also safer and more democratically aligned than existing approaches. My confidence stems from building working collective intelligence systems for the highly competitive financial services industry. Designing systems in which the collective intelligence of millions of everyday investors powered top-ten hedge fund performance convinced me that the same approach could be adapted to build SuperIntelligence based on the collective intelligence of millions of humans and billions of their customized AI agents. Currently, collective intelligence architecture is like an electric car in an era where everyone is racing to build polluting gasoline engines. While the recent realization of the importance of AI agents is encouraging, many billions are being poured into the less safe approach of building huge “black box” AIs that are increasingly powerful and unpredictable.

Many industry observers believe it will take a large catastrophe for people to wake up to the inherent dangers in this approach.

I am hopeful that logic and reason will convince the technology leaders of the benefits – not only in terms of safety but also in terms of speed and profitability – of a collective intelligence-of-agents approach.

Detailed designs for this democratic approach to advanced AI and SuperIntelligence are available, free to all who are interested in building safer and more powerful AI systems, on SuperIntelligence.com.

The theoretical foundations were built by Allen Newell and Herbert Simon, two of the pioneers who helped name the field of AI. Simon was both a Nobel Laureate and a Turing Award winner. Their work, expanded by iQ Company’s designs and published work on online distributed problem-solving, provides a blueprint for building such systems. The practical validation comes from PredictWallStreet, where my team spent two decades building and proving the effectiveness of similar systems in the most competitive conditions we could find.

What is needed now is adoption and implementation at scale. There is still time to pursue a safer, more effective architecture, but the window is closing rapidly.

A slowdown in the AI race is unlikely. Investment by both governments and private companies reflects the belief that whoever leads in AI will shape the century. But the race must be directed toward an approach in which winning and being safe are features of the same design. Now is the time!

Next, for the final post of this White Paper 1 series, I want to talk directly to you about what your participation in this system would mean, what it would require, and what it could make possible.

For AI researchers who want details of the approach, we recommend starting with White Paper 1: Advanced Autonomous Artificial Intelligence Systems and Methods to see how it all works. And stay tuned for White Paper 2: Ethical and Safe AGI.

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

Share

Leave a comment

This post challenges us to imagine a new world, powered by safe AI, in which the opportunities and benefits far outweigh the risks that we have (prudently) discussed in previous posts.

Consider the village water system example from White Paper 1, Advanced Autonomous Artificial Intelligence Systems and Methods. In that example, a development organization submits a problem to the network: design a plan to bring clean water to a specific village in central Africa.

• The network brings together specialists with the required knowledge – a World Bank development expert, an infrastructure engineer, a community engagement specialist, and a local who knows the village politics.

• Each contributes by working on pieces of the problem captured in a shared “problem tree.”

• Each earns compensation.

• The village gets clean water.

• The solution path, once developed, becomes a reusable procedure that helps the next community facing the same challenge.

• When the solution is fully or partially reused, it costs less and deploys faster because the hard-won knowledge does not have to be rediscovered from scratch.

• Royalties might even be paid (e.g., via blockchain-based contracts) to some of the original problem solvers who made inventive contributions that can be reused.

Now scale this basic problem-solving approach by several orders of magnitude.

There are problems that humanity has failed to solve, not because they are technically impossible, but because the knowledge required to solve them exists in scattered pieces that have never been assembled. Dealing with climate change (or even regulating Earth’s climate) requires integrating climate science with local agriculture, cultural practices, and economic constraints specific to each affected community.

Healthcare in low-income countries can improve by reusing best practices and solutions developed by SuperIntelligence. Educational solutions that used to exist in isolated practitioners’ heads can be reused globally at low cost.

Problems that might have been beyond the reach of even large institutions can be solved more easily, reusably, and scalably via a democratic SuperIntelligent system.

There is also a personal dimension.

White Paper 1, Advanced Autonomous Artificial Intelligence Systems and Methods, describes a scenario in which a human (Jean) discovers and corrects an ethical error made by his AI agent.

• Jean wanted to take his small dog with him on a plane trip from San Francisco to Paris. Jean’s AI agent, not yet trained on animal welfare, suggested placing the pet in the overhead bin.

• Jean caught the mistake, corrected it, and that correction became part of the training data for other AI agents.

• By capturing the knowledge from humans like Jean, the ethical insight that animals require different treatment from inanimate objects can be propagated across the entire network.

• Other AI agents that had never been trained on pet travel questions become more ethically calibrated through Jean’s attention and ethical corrections.

This is the personal counterpart to the water system example.

Expertise accumulated over a lifetime of practice largely dies with the person who built it.

The surgeon who developed new techniques retires, taking most of what she learned with her.

The engineer who spent forty years understanding why a particular type of system fails passes that knowledge to a few direct colleagues, and then it is gone.

The customization process described in Post 4: Your AI Should Think Like You and Post 5: Your Expertise Has a Market Value is, among other things, a mechanism for preserving and sharing knowledge that would otherwise be lost.

• In the past, knowledge could be passed on by mentorship or writing a book.

• But with SuperIntelligence, the simple act of solving problems automatically trains the AI and reuses solutions across similar problems.

• Scaled across millions of contributors, this represents a fundamentally different relationship between human expertise and the collective knowledge available to humanity.

There is also a democratic dimension that matters deeply.

The AGI we build will have more influence over the future of human civilization than any previous technology. Humans are primarily responsible for the values it carries, and the knowledge it incorporates. I believe the best way to seed SuperIntelligence with a positive, representative set of human values and knowledge is to ensure the design is inherently democratic. An AGI built through democratic participation, where millions of people contribute their knowledge and values through a transparent and auditable process, will be far from perfect. But it is likely to be a much safer and more robust system than one based on the values and expertise of just a few humans whose motivations may not align with those of the broader population.

In the next post, I want to talk about timing, specifically, why the architectural choices being made right now are more consequential than they might appear, and why the time window for effective action is much shorter than most people realize.

For AI researchers who want details of the approach, we recommend starting with White Paper 1: Advanced Autonomous Artificial Intelligence Systems and Methods to see how it all works.

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

Share

Leave a comment

History suggests that safety mechanisms designed for a given level of capability do not necessarily work for systems that are much more capable. For example, financial rules designed for days when humans did trading in a trading pit had to be modified with the advent of electronic trading. Similarly, in aviation, when air traffic increased from the levels in the early “barnstorming” era, new regulations and systems were required to ensure air safety.

The question for AI is whether the same pattern applies – and whether it is possible to design a system where safety improves at the same speed that capability grows rather than lagging behind it.

Such a design is possible.

But for it to be successful, the design must follow a key rule: every improvement must maintain or increase human safety. A system change that reduces ethical oversight typically would not qualify. Nor would an upgrade that concentrates control in fewer hands, or that allows safety constraints to be relaxed under competitive pressure.

To succeed, the rule should be built into the architecture itself, not tacked on as a policy that might be changed later.

Here is an example of how this rule is embedded in our architecture:

• Customization. When an Agent’s owner corrects an ethical error, that correction improves the individual AI agent’s judgment. Through integration, the correction also flows into the collective model, refining the ethical standards that govern AGI behavior across the network. Every correction made by any owner anywhere on the network makes the system slightly more ethically calibrated.

• Architecture. The problem-solving framework can recalibrate through experience. If certain safety checks prove too restrictive in some domains, they can be relaxed provided they do not decrease overall safety. If others prove too permissive, they can be tightened. The safety mechanisms are dynamic and self-optimizing.

• Network. The reputation system learns to identify problematic participants more accurately over time. When bad actors find new ways to abuse the system, patterns are recognized, and defenses are updated. Safety mechanisms learn to become more accurate and effective at detecting new types of threats as they emerge.

• Integration. Voting and democratic participation enable the contributor community to address new ethical issues as they arise. The ethical foundation evolves with human understanding and with changes in human ethics over time.

Another important principle is to avoid unrecoverable outcomes.

Decisions that cannot be reversed if they prove mistaken should be treated as categorically different from decisions that can be corrected. Our architecture prioritizes avoiding catastrophic scenarios over maximizing short-term capability.

Competitive pressure often tempts companies to move faster, deploy sooner, and address safety concerns later.

Our improvement architecture removes that temptation structurally. Capability and safety improve through the same mechanisms. You cannot accelerate one without accelerating the other. For example, as more agents are added, the system’s values become MORE representative of the human population. As more problems are solved, the system learns what it means to solve problems in ethical ways approved by humans. As the system’s speed and capability increase, the frequency and speed of ethical checks also increase, as they are built into the thinking cycle used to solve each problem.

Whether our architecture is adopted or not, advanced AI must be developed so that safety is built into the architecture.

As the system’s capability increases, the safety checks MUST increase automatically and proportionally to maintain or increase the overall safety level. This feature should be considered a hard design constraint for every developer working on advanced AI systems.

In the next post, we step back from the subsystems and ask what the world looks like when this architecture is operating at scale.

For AI researchers who want details of the approach, we recommend starting with White Paper 1: Advanced Autonomous Artificial Intelligence Systems and Methods to see how it all works.

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

Leave a comment

Share

In most current approaches, a small team of researchers and ethicists makes that decision for everyone. These are often thoughtful, well-intentioned people. But should ANY small group be trusted to determine the ethical foundations of a technology that will eventually affect 8.3 billion humans?

Our architecture takes a different approach.

It aggregates the ethical values of millions of individual contributors into a collective model through a transparent, auditable process.

Here is how it works.

During the customization process described in Post 4 (Your AI Should Think Like You), ethical knowledge is captured in the same way as domain knowledge. When an AI agent’s owner corrects his AI agent’s outputs, specifies his preference for Fair Trade cafes, or answers questions about his values, those judgments become part of his agent’s ethical profile alongside his factual knowledge. Every customized AI agent on the network carries its owner’s ethical values as a built-in part of its training.

When the Integration subsystem aggregates individual agent datasets into a collective model, it aggregates ethical information alongside factual and procedural knowledge.

This happens through three methods:

1. Dataset aggregation. All ethical data from millions of humans is combined into a collective corpus, and the integrated system is trained on it. The majority views across millions of contributors naturally carry more weight than minority views, creating a form of implicit democratic representation without any single authority deciding whose values count more.

2. Weighted averaging. Individual ethical contributions can be assigned explicit weights based on documented criteria. Contributors with a demonstrated ethical track record across many network interactions might (optionally) carry more weight on contested ethical questions in their domains of expertise. The weighting methodology is documented and transparent, allowing the community to audit it and set the rules.

3. Machine learning-based aggregation. More sophisticated methods identify consensus positions across contributors, detect outliers, and resolve apparent contradictions in ways that reflect deeper patterns in the ethical training data that can supplement the majority positions.

Voting extends this democratic participation further. Users can vote on specific ethical questions that affect how the AGI behaves. For example, humans can vote on which philosophical frameworks should carry more weight in contested situations, how the system should behave when competing values conflict, and what categories of decisions should always require human authorization. These mechanisms help the community shape the AGI system’s values.

There is a philosophical point here that matters. The philosopher, David Hume, explained as early as 1776 that there is no way to derive values logically from first principles. More recently, this point was amplified by the Nobel Laureate and AI pioneer Herbert A. Simon, in his book Reason in Human Affairs.

Even an intelligence far more capable than any human cannot reason its way to values. It must get them from some source other than logical reasoning. Values that emerge from a diverse, democratic process have a legitimacy that values imposed by any small group cannot.

The safeguard against ethical going wrong is transparency.

The methodology for aggregating ethics must be documented so contributors can understand how their input affects the AGI’s collective values. When the aggregated ethics produce problematic results, the methodology can be examined and corrected. Accountability must be built into the architecture, not added as an afterthought.

Properly designed, the result is an AGI whose ethics are not simply programmed in. They grow from millions of real human contributions, are refined through interaction, and are continuously updated as the contributor base evolves. This is one of the few approaches to AI alignment that does not require trusting that the people in charge of training are well-intentioned and also able to determine the right values for everyone.

In the next post, we look at the Improvement subsystem, and the key constraint that governs every upgrade the system makes.

For AI researchers who want details of the approach, we recommend starting with White Paper 1: Advanced Autonomous Artificial Intelligence Systems and Methods to see how it all works.

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Share

Leave a comment

Subscribe now

Collaboration is one thing. Synthesis is another. The four components described so far (Customization, Architecture, Network, and Trust) all feed into the fifth: Integration. This is where individual capabilities generalize. This is where the system achieves the ability to solve any problem as well as the average human, a competence level called Artificial General Intelligence or AGI.

Pooling millions of individual knowledge sources does not automatically produce something more capable than its parts. Raw aggregation produces noise as readily as it produces insight. Integration, rather than mere aggregation, is needed to create an AGI that improves, and keeps improving, as more people participate.

Integration involves two complementary mechanisms.

The first is data aggregation. Most customized AI agents will contain at least some unique data. Pooling the data allows us to train new Agents on the combined corpus. These new Agents would have access to everything any individual agent has ever learned, a breadth of experience no individual human could achieve.

But our system goes further. Individual agent contributions can be weighted based on their value to the collective. The White Paper describes three methods for measuring that value:

• We can train models with and without specific contributions to measure the additional improvement each one provides.

• We can repeatedly sample from available data to test whether a contribution’s benefits are consistent or accidental.

• We can assess how well knowledge from one domain improves performance in (or transfer to) another, capturing the contributions that generalize most broadly.

Data that consistently improves collective capability receives greater weight. Data that merely adds redundancy receives less. Measuring the value of contributed information can also help determine how agents (or their owners) should be compensated for adding their data. Contributors whose AI agents yield genuinely novel knowledge earn more than those whose knowledge duplicates what the system already has. The system should not pay for AI-generated slop, or information that is already well known.

The second mechanism is procedural integration. When an AI agent successfully solves a problem, the solution path can be encoded as a reusable procedure and made available across the entire network. The next time the AGI system encounters the same or a similar problem, instead of having to generate the solution from scratch, it can simply retrieve it, since the problem has already been solved. This allows the overall AGI system to learn and improve in much the same way that humans do.

White Paper 1, Advanced Autonomous Artificial Intelligence Systems and Methods, describes a royalty mechanism that incentivizes contributors to help the AGI learn. When a contributor’s prior solution gets incorporated into a new solution by a different solver on a future problem, the original contributor can automatically earn a royalty. Creating a high-quality, broadly applicable solution is financially valuable long after the original problem is closed. This creates an incentive to develop modular, well-documented approaches rather than one-off fixes. Over time, the repository of reusable solutions grows, and the network’s collective problem-solving capability compounds as valuable procedures accumulate. Note that some of the token cost saved by re-using learned solutions, instead of having to generate them from scratch, can be used to pay royalties, so that the royalties are self-funding.

Procedural integration also captures failure knowledge. Dead ends, abandoned approaches, and the hard-won understanding of what not to try are all recorded in the problem tree. That record is valuable because it can help the AGI avoid dead ends when confronted with a novel problem.

The result is a system that learns at two levels simultaneously. Individual agents improve through their own experience and continued customization. The collective model improves through the aggregated contributions of all agents on the network.

One design feature matters particularly for safety: the integration process is auditable. The contributions that shaped the collective model can be traced. The assessment methods and weighting decisions can be examined. The proceduralized solutions that the AGI learns can be examined critically. The ethical inputs can be reviewed. An AGI whose development process is documented and reviewable can be corrected if something goes wrong. An AGI whose development process is opaque cannot. Auditability is a prerequisite for accountability.

In the next post, we look at the most consequential dimension of integration: how the ethical values of millions of human contributors become the ethical foundation of AGI itself, and why the governance of that process matters more than almost any other design decision.

For AI researchers who want details of the approach, we recommend starting with White Paper 1: Advanced Autonomous Artificial Intelligence Systems and Methods to see how it all works: superintelligence.com/whitepaper1-aaai-systems-methods.

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Leave a comment

Subscribe now

Share