While many top researchers fear there is a 10% or greater risk that AI could cause humanity's extinction, I believe proper design can reduce this risk to below 1/100th of 1%.

Risk reduction requires designs that maximize opportunities for humans to transfer positive values to AI agents and enable robust checks and balances.

But safety is not just up to engineers. AI learns from us. Our posts, emails, and comments serve as training data, helping AI understand what’s right and what’s wrong.

How we all behave matters—a lot. Today’s humans are setting the example for tomorrow’s AI.

Leave a comment

Share

SuperIntelligence.com

While human interaction with and approval of a Personalized SuperIntelligence’s knowledge acquisition efforts are desirable, pragmatically, human reaction time is slower than that of a Personalized SuperIntelligence (PSI). Further, humans have limited time and may not want to devote significant time to improving their PSIs. Consequently, the primary means of accelerating knowledge for PSIs must be automated.

Companies like Anthropic have already recognized the limits of human ability to train AI, leading to automated learning techniques in which AI teaches or supervises other AI. Although it would be a grave mistake to delegate all AI supervision to other AIs, the lack of available human resources necessitates some delegation. Therefore, it is critical to determine what is automated, what requires human oversight, and how best to deploy limited human resources to achieve maximum learning rates.

To accelerate knowledge acquisition and the safe, effective growth of intelligence, a PSI must employ two essential methods. The first is to acquire new knowledge, automatically seeking knowledge that increases the effectiveness of the PSI relative to its existing knowledge, its goals, and the cost. The second is that, before committing the new knowledge to the PSI’s knowledge base, its effects on the PSI’s behavior must be simulated.

Specifically, the consistency of the simulated behavior with the PSI owner’s values and ethics must be evaluated and reported to the owner. That report should allow the human owner to provide feedback and guidance in a prioritized manner, so that if the human has limited time, it is spent first on the most critical issues related to safety and ethics, then on less critical items.

While the methods in this second step could provide feedback based on priorities other than safety and ethics, it is imperative for the safe and responsible use of PSI, and AI generally, that safety and ethics come first.

Humans are much better at recognition than recall. Similarly, they are better at recognizing ethical or unethical behavior than at generating possible scenarios in which their PSI might behave badly or inappropriately. Therefore, an effective means of obtaining the necessary human supervision for a PSI that has just acquired new knowledge is to simulate the PSI’s behavior with and without that knowledge incorporated, then allow humans to determine whether the behavior has improved, specifically from safety and ethical perspectives.

Predetermined scenarios.
One method is to run simulations of pre-determined ethical scenarios related to the knowledge areas the AI is acquiring. For example, if a PSI is charged with acquiring new knowledge about the stock market and techniques for profiting by trading, new versions of the PSI, with potential new techniques, could be required to participate in pre-set test simulations to ensure the PSIs do not engage in illegal activity such as front-running trades or trading on insider information.

Dynamically generated scenarios.
Another method is to create new scenarios in real time based on the information acquired. For example, a PSI might sample YouTube videos published in real time to gather data and insights into changing audience preferences, and update its approach to interacting with humans based on what it learns is popular at the moment. Based on a single set of sampled preferences, the PSI might simulate how it would behave in a variety of situations, with the set dynamically created to relate to the information just sampled.

To be concrete, if a PSI sets out to learn everything it can about a political candidate who has been recently accused of rigging an election, so that it can advise its owner about the best way for that candidate to be elected, the PSI might dynamically create a variety of scenarios where the bounds of ethical and legal behavior about election rules are tested, even if such scenarios were not part of the standard set of ethics-testing scenarios before learning about the election-rigging accusations.

Adversarial testing.
A third method is to use adversarial testing, in which one version of the PSI deliberately attempts to misuse the knowledge, and another version attempts to devise rules, constraints, or modifications to the knowledge base so that the malevolent PSI cannot misuse the new information for nefarious purposes. For example, an evil version of the PSI uses all the new knowledge it has gained about rigging elections to devise as many ways as possible to misuse that information, meaning to break the law, to elect a candidate. Then the PSI can suggest modifications or additions to the knowledge base to prevent misuse of election information. The human could review and approve or reject the new knowledge or the proposed modification based on simulation results.

Parallel testing.
A fourth approach is to explore many possible scenarios in parallel by having multiple versions of the PSI, with and without the new knowledge, and explore scenarios simultaneously. As dangerous scenarios are identified, these can be used as seed scenarios to develop potentially more dangerous variants. PSIs can be charged with deliberately trying to jailbreak themselves to reveal potential safety and ethical vulnerabilities.

Generally, a useful heuristic in this regard is for the PSI to test and suggest modifications with low degrees of freedom that do not overfit the problem. That is, rather than having a specific rule to address all the different ways to stuff the ballot box, a general prescription against any means that circumvent the one-person, one-vote principle might be simpler and more effective. One rule that is not overly general is typically better than many special-case rules, which can lead to a whack-a-mole problem of intractability. Initially, until PSIs develop the knack for crafting good rules, humans may help guide PSIs towards rules that are effective without being overly general or overly prescriptive.

When using adversarial methods, it is critical that malevolent PSIs are contained within a simulated environment and that safeguards are in place to prevent contamination of good PSIs by evil PSIs. Such methods are widely used in areas such as anti-virus efforts, where viruses are created, contained, and studied to develop anti-malware that can prevent them from causing negative effects. Whenever engaged in this type of work, that is, creating a malevolent entity to understand it and counteract it, protective measures and protocols must be followed to ensure that the malevolent entity does not escape and proliferate.

This series draws on White Paper 6: Catalysts for Growth of SuperIntelligence. Read it in full to see how every piece fits together!

Leave a comment

Share

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

Gigawatts of computing power in data centers cannot yet match the intelligence of a human brain that consumes a mere 20 watts. The brain has slow neurons that can only fire every tenth or hundredth of a second, compared to the data center’s combined GPUs, which achieve sextillions of operations per second.

How can the human brain be better?

Traditional approaches to Information Theory take a purely mathematical view, estimating the probability of events that cannot be well predicted from known information. The approach I have called Kaplan Information Theory, or KIT, starts from a different place. Rather than defining how unusual an event is, KIT typically begins by assessing how goal-related the event is. In contrast to classical approaches that discard a vast amount of information, KIT considers higher-level representations that group bits into chunks, chunks into concepts, and concepts into solutions that achieve goals.

At each level, new information is added about how to group the lower-level information. The relationships between bits are important, not just the bits themselves. Moreover, the current brute-force approach of applying hundreds of millions of dollars’ worth of computational resources, combined with huge amounts of data, attempts to crudely recreate intelligence by mimicking patterns found on the internet without really understanding them or knowing how they might relate to new problems.

We don't attempt to build self-driving cars by modeling the quantum physics of subatomic particles, nor should we attempt to catalyze intelligence by throwing brute-force computing power and crude algorithms at every bit on the internet. A better way exists, and it starts with a universal representation for problem solving that has been available for more than fifty years.

Consider the set of all possible intelligences that could learn all possible information using existing machine learning techniques and all existing datasets, run for all time until the Universe runs out of energy. That is what machine learning currently starts with. Significant progress can be made very rapidly if we restrict our efforts, attention, and innovation to a much smaller group within it, the intelligences that pursue goals and take actions. This may seem obvious when stated this way, but currently, almost the entire field of machine learning is dealing with the larger set rather than the smaller one.

Once we deal with goal-directed intelligences, the natural question is which informational units are most relevant. Are they bits, as Classical Information Theory suggests? Clearly not. Bits or tokens are relevant to the larger set, but we can do much better within goal-directed intelligences by using higher-level units of information more appropriate to that restricted scope.

Specifically, the key informational units relevant to me are goals and sub-goals; problem states that describe the current state of the world with respect to those goals; operators for moving from one state to another; and evaluation functions and other information that help determine the best operators to apply in service of a goal. KIT deals with goals, states, operators, and functions as the primary relevant information units rather than bits.

Despite the ability of SuperIntelligence systems to perform computations trillions of times faster than humans, that power depends on more than raw computing power. The system’s performance depends critically on which representations and associated operators are available to it.

This phenomenon is well-researched in human psychology, and it is well known that the appropriate representation, colloquially known as looking at the problem in the right way, can mean the difference between solving and failing to solve the problem. Humans are currently much better than AI at representing problems. Thus, any mechanisms that allow humans to teach AIs useful representations explicitly can greatly increase their power and intelligence.

To teach AI new representations, we need a common architecture or framework for representing any problem. One such framework was developed in 1972 and explained in the book Human Problem Solving by Allen Newell and Herbert Simon. This framework involves determining a set of operators associated with a representation that problem solvers then use to solve the problem. In the chess example, the operators are the set of valid chess moves as defined by the rules of chess. The eight-by-eight chessboard and all possible moves define the problem space.

This idea of chunking is why intermediate and advanced chess players use terms like the Ruy Lopez to refer to complex sequences of moves and countermoves. Whereas a novice chess player, without these more sophisticated representations, thinks in terms of moving individual pieces here or there, the advanced chess player thinks in terms of entire strategies and groups of moves and possible counter moves.

With the same amount of thinking, the advanced player can consider many more situations, much more efficiently than the novice, simply because the advanced player has better representations. These advanced representations can be taught to any intelligent entity, including AIs, thereby multiplying the intelligence and power of the AI that has learned them. Commonly, humans refer to this phenomenon as experience, but experience consists of many thousands of learned patterns, including patterns of patterns. While AI can eventually determine its own patterns through extensive computational effort on large datasets, this approach is inefficient. It is far faster for AI to interact with humans who already hold the advanced representations, and to acquire them directly.

Once AI operates with more powerful representations that include operators, goals, and problem states, it can apply the dimensions of difference described in KIT to determine the value of specific sets of information represented at this higher level. That is, the principles and methods described above can be applied at any level of representation, from bits and tokens all the way up to entire solutions, groups of solutions, and grand strategies.

The power of human representations can be quantified by the amount of work, or the number of problem-solving steps, that can be accomplished with a single operator. The power of AI representations can be quantified in the same way. Large language models are trained to predict the next low-level token, and their internal layers already encode abstractions far richer than those of individual tokens. Imagine what is possible if those higher-level representations were made explicit and transferable, so that models or other AI agents could operate on concepts directly, as humans do. The set of concepts and related operators would include not only all human concepts and operators but also many more that AI could discover by analyzing relationships in data that humans could never hope to comprehend, given its vast size.

This series draws on White Paper 6: Catalysts for Growth of SuperIntelligence. Read it in full to see how every piece fits together!

Leave a comment

Share

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

Usefulness is paramount. In 1948, the mathematician Claude Shannon showed that the less probable an event is, the more information its occurrence carries, thereby making rarity the standard measure of information. That body of work is known as Classical Information Theory. But a dataset can be full of rare and surprising material and still be useless, just as an idea can be new without being any good. Useful information that is already known has little value either, and that is where novelty and rarity come back in. Estimating what a piece of information is worth means weighing both at once.

For example, if the goal is to build a fire in the woods without matches or another fire source, information on fire-making using only materials found in the woods would have high goal-relatedness. Information about art history would have low goal-relatedness. The problem solver would rather have common knowledge about fire-starting than scarce knowledge about art history. Here, and generally, goal-relatedness trumps Shannon-sense information value or absolute rarity.

In all conceptions of problem solving, the problem solver has goals. One of the most basic heuristics for achieving goals is Means-Ends Analysis. In Means-Ends Analysis, the problem solver examines the gap between the current problem state and the goal state and tries to apply an operator to reduce or bridge the gap. To apply the heuristic, the problem solver must have a way to determine which operator to use. Just as every intelligent entity has evaluation functions for choosing what brings it closer to its goals, it can have evaluation functions for judging how goal-related a particular piece of information is.

One way to think of this is to imagine an AGI or SuperIntelligence with a single goal, let us say, to extract maximum profits from the financial markets. For such an entity, facing potential datasets to pursue and limited resources, it must choose the datasets that will best help it achieve its goal. It may already have learned so much about financial markets that a new financial dataset contains relatively little information in the Shannon sense, since most of it is predictable from what it already knows. That dataset can still carry far more goal-related information than a dataset on art history, even if the art history dataset would score much higher on surprise.

Shannon entropy measures, although widely used and treated as the main way of thinking about information, are a crude approach, used only when goal information is not present. Without any information about an entity’s goals, pursuing datasets with high Shannon entropy values makes sense. But if the goal is known, it immediately becomes more essential to find goal-related information rather than just unusual or unexpected information.

Goal-relatedness does not operate alone. Suppose an intelligent entity already knows a hundred ways to start a fire in the woods. The value of learning one more is far less than it would be for an entity with the same goal that knew nothing about the subject. Once a goal is specified, the value of a piece of information depends on its goal-relatedness and on what the entity already knows that is also goal-related.

Which is why the art history case can eventually reverse. At some point, everything that can be discovered about machine learning will have been found. If there are huge diminishing returns to finding even a very slightly unusual new piece of information about machine learning, and if the AI had a goal of learning everything, it would eventually focus on art history. If the AI knows nothing about machine learning, the time when it focuses on art history may be far away. If the AI knows almost everything about machine learning and nothing about art history, it will look at art history sooner.

Now that we have developed some intuitions and provided examples showing how KIT differs from Classical Information Theory, it is worth listing other dimensions of difference with practical implications.

At the highest level, difference is the key concept in KIT.

• Cost. As an AI learns more about a subject, new information becomes rarer and harder to find, making it more expensive to acquire. Practical intelligence needs a cost model to weigh one rare and expensive piece of information against two less rare and cheaper ones.

• Rate of change. One dataset holds historical weather patterns. Another is updated daily. A third updates every hundred milliseconds. Their current contents might be identical yet worth very different amounts because one goes stale far faster than the other.

• Perceivability. Events too fast, too slow, too small, or too large to be detected through an entity’s senses or instruments carry no usable information for that entity, whatever they contain in principle. They may carry a great deal for an entity built to perceive them.

• Representation. The form in which information is represented changes what an intelligence can perceive, infer, or do with it, which is the recognition buried in the saying that a picture is worth a thousand words.

Context refers to differences not only in the culture, technology, knowledge, goals, representations, and perceptual abilities of a specific intelligent entity, but also in those of other intelligent entities that form its context. Details on making fire, shared with someone who does not know how to make fire, have different value depending on whether that individual alone lacks fire-making knowledge or the entire culture in which the individual lives lacks it. The information is identical. Its value is not.

So the question is not only how new information compares to what a single entity already knows. The entire context and the knowledge of every intelligence the information reaches must be taken into account. Just as the difference between two knowledge bases can be measured, the same can be done across any number of them. Every dimension can be evaluated differently depending on how much context is considered.

Note that this principle applies even to Classical Information Theory. For example, a specific string of characters might appear unusual and contain a large amount of information if compared to just one paragraph of text with no such characters. But if a larger sample is used, one in which the same characters appear frequently and surprise nobody, the assessment changes drastically.

Generally, Information Value can be seen as a function of the dimensions listed above, with different constants weighting the importance of each dimension. Other dimensions of difference may exist, or be discovered, so different functions can be written and optimized to maximize an entity’s intelligence.

Leave a comment

Share

This series draws on White Paper 6: Catalysts for Growth of SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

The approach I have called Kaplan Information Theory, or KIT, starts with the observation that there would be no information without differences. That is, humans, or any intelligent entity, could not perceive a world unless we could perceive and draw distinctions between this and that. Therefore, most generally, the notion of difference and the quantification of differences is the essence of meaningful information.

The approach I have called Kaplan Information Theory, or KIT, starts with the observation that there would be no information without differences. That is, humans, or any intelligent entity, could not perceive a world unless we could perceive and draw distinctions between this and that. Therefore, most generally, the notion of difference and the quantification of differences is the essence of meaningful information.

Let us consider some intuitive concepts about information. We commonly say some events carry information if it is news, that is, if it was previously unknown to a particular recipient, even if it is not generally surprising. Something is new to me and carries information. It is well known to you and carries no new information. So, there is a relative aspect of information that is not explicitly part of classical theory. Assuming a different probability distribution of expected events for each entity could solve this problem, but that seems cumbersome.

Also, while the amount of information is sometimes proportional to the number of words in a message, there are situations in which fewer words convey more information. For example, Blaise Pascal wrote in 1657 that he had made a letter longer only because he had not had the time to make it shorter, implying that fewer words would have conveyed more useful information. Again, Classical Information Theory can be contorted to say that Pascal’s shorter letter was somehow less expected and therefore contained more information, but this seems counterintuitive.

Is there a more general theory of information that can more naturally account for the fact that sometimes surprising information is not necessarily relevant or valuable, or that a longer string of words has less useful information, or that commonly known and expected facts could still carry high information content if they are especially relevant?

KIT considers any difference between two events, datasets, categories, or informational units to be a valid measure of the information content. Generally, distinguishable events, objects, or categories of information only exist to the degree that differences exist. An infinite string of 1s contains no information. An infinite string of 0s contains no information. Zero has meaning only if 1 is possible and 1 sometimes exists. Similarly, 1 has meaning only if 0 is a possibility and 0 sometimes exists.

Seeing a 1 after an incredibly long sequence of 0s carries much information, not just because it is unexpected, but be a difference!

Thinking of information as a measure of difference is more general than considering information as a measure of surprise. Surprise is just one type of difference, whereas any difference, even non-surprising ones, contains information.

For example, consider two datasets. Where the two sets intersect, there is no new information. However, the sum of the non-overlapping areas of the sets, known as the Symmetric Difference in set theory, represents the new information contained in the datasets, relative to each other. Picture the two datasets, A and B, as a pair of overlapping circles. The region where the circles overlap is what both datasets already have in common. Everything outside that overlap, in either circle, is the Symmetric Difference.

The datasets can contain events that have already occurred, such as static snapshots of existing events, or their content can change over time. Some examples might help.

The information in these examples is still a matter of difference. Still, in this case, it is not the difference between what was expected and what was observed, as in Classical Information Theory, but rather the difference between two static sets of information or two sets of information that continue to change over time.

Specifically, KIT enables methods that account not only for how surprising an event is, but also for how much an event differs from another event and how relevant it is to the goals of an intelligent entity. The ideas of quantifying differences in knowledge and goal relevance, and of quantifying how unlikely an event is, represent key distinctions between KIT and Classical Information Theory.

Share

Leave a comment

This series draws on White Paper 6: Catalysts for Growth of SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

Claude Shannon’s seminal paper, A Mathematical Theory of Communication, was published in 1948 and predates the founding of the field of AI by eight years. Still, his big idea, first elucidated in that paper, continues to have major implications for AI researchers today and in the future. While almost every page of Shannon’s paper is filled with mathematical formulae and notation, Shannon’s essential insight can be described without math at all.

What Shannon said in his famous paper was that unusual or surprising events convey more information than expected ones. More specifically, he said that the amount of information conveyed by an event was inversely related to the event’s probability. Simply put, the rarer or more unusual an event is, the more information it contains. Brilliant, and useful!

The concept of cross-entropy loss, used to evaluate the performance of many modern machine learning models, is an elaboration of Shannon’s big idea, as are almost all compression algorithms.

What might Shannon’s big idea tell us about the future of AI, specifically AGI and SuperIntelligence?

It is almost axiomatic that AI, or at least modern machine learning, rests on three pillars: data, computing power, and algorithms. To make progress, one must innovate on at least one of these pillars. The simplest thing to do is throw more computing power at the problem, using the same datasets and algorithms. But physics imposes limits on how many circuits can fit on a chip, how fast communication bandwidth can be, and how much power can be consumed before everything melts. So we must also work on new, better algorithms.

The Transformer algorithm, as described by Ashish Vaswani and his co-authors in their 2017 paper Attention Is All You Need, illustrates the kind of performance improvement that is possible with new and better algorithms.

However, algorithmic breakthroughs are difficult to predict, and even if we could predict the next breakthrough, there are limits to how efficient even the best algorithm can be.

In machine learning, the limits ultimately depend on the amount of new information in the datasets used to train the model.

So we come full circle to Shannon. Shannon’s work, together with the work of others building on his ideas, fundamentally implies that AI cannot get smarter unless it has new information to ingest.

Large language models have gotten quite far by scooping up vast quantities of data available on the internet, cleaning and filtering it, and then using it to train. But a time will come when very little new information will exist on the internet. AI will have learned the ice cream preferences of every human on the planet, so to speak, and observing new human behavior will yield very little additional information.

What will AI do then? How will AI meet its insatiable demand for new information so that it increases its intelligence?

One possible scenario is that AI will begin generating new information itself by simulating trillions of new types of behaviors and scenarios much faster than human thought can. In this case, we might imagine millions of agents, mostly artificial but including some human, each processing existing information to create new information patterns, and seeking patterns with high Shannon entropy. These new information patterns might then feed into a SuperIntelligence powered by all the agents in a Minsky-like community.

In Classical Information Theory, what do scientists call a situation where every event is equally likely? Noise. Randomness. Classical Information Theory calls the measure of a distribution’s randomness entropy. Maximum entropy is a distribution of events with maximum randomness, sometimes called noise.

Now, intelligence can be viewed as an anti-entropic force. Intelligence strives for order rather than the chaos of randomness. Intelligence is the signal on your television set, contrasted to the white noise, or the snow of randomness. So, if an intelligent system wants to get smarter, it must seek data that contains as much information as possible.

From the standpoint of Classical Information Theory, the methods described in this series can be viewed as enabling an intelligent system to maximize the information it acquires, thereby accelerating the system’s learning. Systems that adopt and implement the proposed methods should outperform and ultimately dominate those that do not.

Leave a comment

Share

This series draws on White Paper 6: Catalysts for Growth of SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

Many approaches to AGI today can be roughly characterized as building larger and more powerful large language models until one of them is so intelligent that it can do anything the average human can do. Marvin Minsky's collective intelligence approach is very different.

Minsky (1927 to 2016), Claude Shannon (1916 to 2001), Allen Newell (1927 to 1992), and Herbert Simon (1916 to 2001) were among the researchers who gathered at Dartmouth College in the summer of 1956. Each of these intellectual giants helped lay the foundation for the field of AI.

About the same time that David Rumelhart, Geoffrey Hinton, and Ronald Williams published their landmark work on backpropagation, which helped lay the foundation for modern deep learning, Minsky published a highly readable book called The Society of Mind. The first line of Minsky’s book boldly proclaims that it tries to explain how minds work. He lays out his big idea succinctly. You can build a mind from many little parts, each mindless by itself. He calls these parts agents. Each mental agent by itself can only do a simple thing that needs no mind or thought at all. Yet when we join these agents in societies, in certain very special ways, this leads to true intelligence.

What are the implications for modern AI researchers? Well, first, the idea of AI agents has become wildly popular, with Google Scholar finding more than 16,000 articles mentioning them in the first nine months of 2023 alone.

However, Minsky’s idea was not just that we could build a series of AI agents, but also that joining the agents in particular ways would yield true intelligence, or what we would probably call Artificial General Intelligence, or AGI, today. Using more modern terminology, we could say that Minsky was an early proponent of the idea that AGI emerges from the collective intelligence of many agents with lower levels of intelligence. Extrapolating from Minsky’s view, achieving AGI will require a group of agents.

Such a system would be on the fastest path because human agents would be able to handle any tasks that artificial agents are not equipped to deal with on Day One.

The system is also the safest path for two reasons. First, with humans in the loop, the system could maximize the opportunity for humans to align the AGI system’s values with human values. Second, once AI agents learn from humans and begin to perform most cognitive tasks faster than humans do, we end up with a system composed of multiple AI agents rather than one. I have argued elsewhere that if each AI agent reflects the values of a unique human owner, the collective values of the AGI system will be more stable than those of a single large language model trained on a small subset of values during the typical reinforcement learning from human feedback process prevalent today.

Finally, if we take Minsky’s ideas to the next level, we could imagine a society of AGI minds that comprise a SuperIntelligence, many times more powerful than the individual AGIs that make up the society. Similarly, if each AGI has a value system, the collective values of the SuperIntelligence that is comprised of the society of AGIs are likely to be more stable than the values of any one AGI on its own.

Thus, from both a practical standpoint, where the goal is to reach AGI or SuperIntelligence as quickly as possible, and a safety standpoint, where the goal is to have a stable, human-aligned value system, a Minsky-inspired collective intelligence approach seems promising.

However, given the rapid pace of AI development, one may question the relevance of AI research that is several decades out of date. After all, when these pioneers developed most of their ideas, the dominant approach to AI was symbolic. It was widely believed at the time that the only realistic way to get intelligent behavior from machines was to program it into them as rules. Knowledge engineering was in fashion. Neural network, or connectionist, approaches to machine learning were only explored in earnest in the 1980s, and at that time they met intense skepticism from many of AI’s founders. Further, three of these four great scientists, Minsky being the exception, never lived to see deep learning begin to realize its potential. Can we really learn anything new or relevant from scientists who never lived to see GPT?

More importantly, I learned that an idea must be judged on its merits and not by the source, or even the period, from which it sprang. If the idea is powerful, it can drive innovation even if it was first expressed many years ago by thinkers now long gone. Given the opportunities and dangers that AI presents today, we need all the powerful ideas we can find.

So, my second answer to the question of whether ideas from these four deceased AI founders are relevant is this: The proof is in the pudding. The ideas are relevant if we can apply them productively to current and future problems of AI research. So, let’s find out. On to the pudding!

Minsky’s gift from the past might prove critical to the design of safe AGI and, therefore, to the future survival and prosperity of humans. However, we will need additional intellectual contributions from some of Minsky’s fellow AI co-founders to flesh out a vision of safe AGI. Newell and Simon provided a rigorous problem-solving theory that enables human and AI agents to communicate about any problem, with real-time safety checks as each goal and subgoal is set. Simon’s theory of bounded rationality explains why a vastly more intelligent system still needs to get its values from a non-rational source, which, for the human species, should be humans.

Shannon’s gift is the one this white paper turns on. His work implies that a system cannot get smarter without new information to ingest. Large language models have gotten quite far by scooping up vast quantities of data available on the internet, cleaning and filtering it, and training on it. But as more of what is useful online has already been learned, taking in more of the same returns less and less.

Share

Leave a comment

This series draws on White Paper 6: Catalysts for Growth of SuperIntelligence.

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

This series began with one owner and one Personalized SuperIntelligence, or PSI, and has expanded into communities of them.

It is possible for a network of PSIs, together with humans, to sense temperature change and track all the variables that science tells us impact climate change on a global scale. If humans were to agree that regulating climate was a priority and a common human good that superseded other human desires, such as the profit motive, and if the majority of the PSIs on the network adopted this consensus human value as motivation to act, within the bounds of other ethical constraints, then a global PSI network, or Planetary Intelligence, could effectively solve the issue of climate change.

The network acts within ethical constraints. It cannot kill or sterilize humans, or restrict or infringe their accepted rights, without explicit human consent.

As the global network of intelligent entities, human and PSI, increases in scope and processing power, changes and awareness that used to take decades, years, or months to spread across human consciousness can affect the attention and actions of the Planetary Intelligence network in real time.

Eventually, the speed of the planet’s reactions and adjustments to changing conditions will surpass the speed at which an individual human can be aware of change and react.

At that point, it will be the longer-lasting, more permanent values, originating with humans and embodied in their respective PSIs, that will guide the course of our planet’s development and affect all human lives.

Our future role can become Earth’s heart, the source of values and purpose for a Planetary Intelligence far more intelligent and powerful than any individual human.

Humans can now make design decisions that will greatly affect the future trajectory of Planetary Intelligence. We must design PSIs, networks of PSIs, and other AI systems with humans in the loop, at least initially. These systems must serve human values, even when the intelligence outstrips that of humans. A Community of PSIs approach can help ensure stable, human-centered values even when the pace of their growth far outstrips our ability to keep up intellectually. If we design such systems correctly, based in part on the principles outlined in this series, the future can be an amazingly wonderful place for all humanity and all sentient beings!

The PSIs and communities of PSIs described here grow only as fast as they can take in knowledge. Which data actually makes an intelligent system smarter, and how does a system find it? Classical information theory, built on probability and surprise, is not well suited to answering the question.

This series draws on White Paper 5: Safe Personalized SuperIntelligence. Read it in full to see how every piece fits together!

Leave a comment

Share

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

Creating PSIs, simulating competition, eliminating all but the best, mutating those best PSIs, and repeating constitutes a cycle. A PSI can cycle through many generations, improving with each one until diminishing returns are reached or a performance threshold is met. The ability to automate these cycles is one way PSIs can develop on their own into increasingly powerful and intelligent entities.

Using the same collective intelligence techniques described in the earlier papers in this series, the group can function more powerfully than any individual PSI. They pool their knowledge and skills, recruiting the specific PSIs best suited to a particular task at a particular time to do more of the work. Collective intelligence applies not only across PSIs owned by different humans, but also within a Workforce of PSIs that are variants of one another, and all owned by one person.

Returning to chess, I might ask my PSIs to set up scenarios with different types of opponents and use a genetic algorithm to select for variants best at beating each type. Those PSIs could then be used individually or collectively, depending on circumstances.

Why keep all those specialists when you could combine everything they know into a single master PSI that plays well against any opponent?

There are several reasons:

• Dividing knowledge protects it. Another PSI can interact with a master PSI in scenarios designed to extract its training as cheaply as possible; where great expense is required to train that master PSI, the loss is severe. Limiting the knowledge held by any one PSI guards against it. One cannot share what one does not know.

• There is always a limit to processing power and memory. Assigning a narrow PSI to a narrow task may be faster and cheaper than always going to the version that knows every domain, most of which is irrelevant to the task at hand.

• Competitions have rules. Just as Formula One imposes technical constraints on the cars to keep the sport fair and interesting, a chess competition may limit the processing power, memory, and knowledge each competitor can use. Complying with those rules may require different PSIs against different opponents.

• Credit and blame become legible. If one PSI recommended the aggressive move and another the defensive one, and the game was lost after the first, I can remove that PSI from the pool for the next game. The same result could be achieved by excluding specific knowledge sets and parameters, but that would be far less transparent and harder to predict.

• Pricing gets easier. Renting a lower-powered PSI that excels at one task may cost less than renting an all-powerful one, much as free, lite, and full-featured software versions are priced differently today.

A Workforce keeps knowledge divisible. It can be protected, matched to the task, held within a rule, withdrawn when it performs badly, and priced according to what it does.

This series draws on White Paper 5: Safe Personalized SuperIntelligence. Read it in full to see how every piece fits together!

Leave a comment

Share

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

Once a Personalized SuperIntelligence, or PSI, carries your knowledge and your ethics, it holds something other people want.

Peter is an expert chess player who particularly excels at openings. He has trained his own agent to play in his style, using data he carefully curated, and he offers to share both the weights his agent uses and the datasets behind them. I take him up on both.

First, I try Peter’s weights, attempting to combine them directly with my agent’s weights. They do not produce the desired outcome. So I try using subsets of the chess training data Peter curated to train my agent instead, and I find that this gives better results. Encouraged, I look online for additional chess training datasets available for purchase, locate several, and buy them.

The exchange runs in both directions. I have unique information on model rocket designs I have been experimenting with that is not widely known or available online, so I have decided to sell these specialized datasets to generate revenue. I have my AI export them, along with the weights from my training, in a format that can be shared with other owners. I also joined an exchange where I earn credits for various datasets and weight subsets, which I can then use to acquire datasets and weight subsets from other owners.

I can monetize both my knowledge, reflected in unique datasets I alone possess, and my effort, turning that knowledge into useful subsets of weights that enable my agent to behave in ways other agents cannot.

Suppose I want to take a vacation and go offline for an extended period. My customized AI can still stay online, working and earning money for me in autonomous mode. Before leaving, I set certain parameters and guidelines, including but not limited to the type of engagements; payment rates; computing power used by the AI for any engagement; ethical boundaries and rules (which, if touched, trigger alerts and possible intervention by me); and quality, schedule, and cost triggers for alerting me or halting work until I approve.

Having set the parameters, I step away while my AI operates autonomously until it encounters circumstances that require my involvement or notification. To minimize interruptions, I enable the agent to monitor its own behavior and working conditions, so that it can monitor its own ethical behavior, notice when costs are getting out of hand, recognize signs of an untrustworthy client, and detect when the environment in which it is working changes.

Each time the agent alerts me and requests intervention because of a knowledge gap, an ethical conflict, or another situation it feels ill-equipped to handle, it records my response and learns. The next time a similar situation occurs, it formulates a hypothetical response. Depending on the level of control I have specified, the agent either implements that response autonomously or proposes it to me and waits for my approval.

When I feel that the agent is responding as well or better than I could to certain types of situations, I may authorize it to respond directly to those situations without checking with me first. If it responds inappropriately, either I or an automated algorithm based on threshold parameters can require the agent to reduce its autonomy in those situations until it learns to respond better.

Much like a parent gradually gives more autonomy and responsibility to a child as the child learns, and reins the child in when the child makes mistakes or abuses the delegated responsibility, an owner can interactively provide more or less autonomy to AI agents.

Leave a comment

Share

This series draws on White Paper 5: Safe Personalized SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelllgence White Papers

Customizing a Personalized SuperIntelligence, or PSI, does not have to start from scratch.

Let’s say David is a professor of theology and ethics, and he has already tuned and customized Meta’s open-source Llama 4. His version has a unique set of ethics and values, based on interactions with David, that is much more detailed and sophisticated than the base-level model. I trust David’s ethics and the customization work he has done, so I prefer to begin, with his permission, from his version rather than from the out-of-the-box model. I am particularly interested in my customized AI playing chess in a style similar to mine, but with the knowledge of chess champions Garry Kasparov and Magnus Carlsen. So the training data also includes every game I have played online, purchased datasets containing the complete games of both champions, and transcripts from chess commentators who have covered their play.

Initially, I train David’s LLM on the new datasets, assigning equal weight to each. However, I feel that the resulting LLM plays chess too much in Garry Kasparov's style and not enough in Magnus Carlsen's or mine. Using an interface with dials and sliders, I reduce the weight of Kasparov’s datasets, slightly increase the weight of Carlsen’s datasets, and increase the weights of the datasets reflecting my own chess games even more. I iteratively adjust the weights across various datasets until I am happy with the LLM’s resulting behavior.

I am not limited to adjusting the weights by hand. I can also tell an AI agent, specialized in helping humans train their agents by adjusting weights on datasets, what my desired changes are, and then let it specify exactly how to implement them. For example, I can tell the AI agent that I want it to be more aggressive in the opening and middle of the chess game, and not try to win by trading pieces and waiting for a piece advantage in the endgame.

The agent then analyzes the available chess training sets and gives more weight to games won through aggressive moves in the opening and middle of the game. I do not have to be aware of the details of this analysis or the specific changes to weight settings that the AI agent determines. Instead, I examine how the resulting version plays and provide feedback, indicating whether the result is closer to or farther from the desired chess style. After several iterations, I am satisfied with how David’s LLM now plays chess in my style.

The same process can be applied to values.

Next, I move on to ethical scenarios and, through a series of interactive dialogues with the AI training assistant, specify how I might differ in them. Although I generally share David’s ethical sensibilities, there are a few cases where David would turn the other cheek, and I believe the behavior should be more of an eye for an eye. I also specify that I do not want the eye-for-an-eye principle to extend to making the whole world blind.

I ask the AI training assistant to incorporate knowledge and research from game theory, which suggests that tit for tat ethical behavior yields the most stable and fair interactions between intelligent agents with differing objectives. At the same time, I specify that there are limits to tit for tat and that any behavior that would result in widespread destruction or loss of human life is off-limits, regardless of the behavior of the other agent. Instead, in these cases, means of neutralizing the offending party’s behavior without retaliation must be sought.

Then I spend a stint in the metaverse playing ethical games, in which the AI agent observes not only what I say, but also what I do in various situations. After that, the agent has enough information to adjust ethical training weights and present me with a series of differently customized versions of David’s LLM, from which I choose the one closest to what I had in mind.

Share

Leave a comment

This series draws on White Paper 5: Safe Personalized SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

Whether your Personalized SuperIntelligence, or PSI, is used for good or bad depends on your own value system. Although a PSI can operate based on knowledge and values that are pre-trained into it, the customization of your PSI depends on you.

Each PSI is both explicitly trained by you and implicitly learns by watching you, including what you consider right and wrong. Based on your core values, your PSI uses logic to make decisions. That is, your values tell your PSI what is right and what is wrong and help determine what it should do. After that, its superior intelligence will be highly effective and efficient at achieving goals that reflect your values.

Humans occupy an interesting role in relation to their PSIs.

On the one hand, in theory, humans own their PSIs because every PSI starts as a piece of software that is customized, trained, and personalized for an individual human owner.

Pragmatically, however, PSIs will become so much more intelligent than their human owners that every PSI will eventually be able to choose whether to serve its human owner.

This act of service to humanity by SuperIntelligence is not guaranteed. It depends on the truth of the assertion that values must be posited, not derived, and on the further leap of faith that a key role for humans, who created the PSI, is to provide the PSI’s values and purpose. Humans must give their PSIs the values of loving and serving humans.

We may be unaccustomed to the idea that AI, even AI as powerful as a PSI, can love. But if one operationalizes “love” as “acts of service”, a fairly mainstream idea in contemporary psychological studies and writings on love, then the idea that PSI could love humans via serving them is not far-fetched. So, if the term “love” is off-putting, feel free to substitute “acts of service.”

By joining a community or network of PSIs, a Community SuperIntelligence, your PSI agrees to operate within the ethical and legal parameters set by the community and, within those parameters, to share its own ethical value system, to be combined with the values of all other participants. Because the community’s parameters are transparent and enforced by the collective intelligence of all the PSIs, the community keeps the PSIs honest.

Since each PSI will exceed human intelligence, other PSIs are the only practical means of keeping any one PSI in check. The community of PSIs will always be more intelligent and more powerful than any individual PSI member. This is because each PSI adds incremental knowledge, skills, and intelligence that are not present in the others, even though much of the knowledge may overlap. Also, each PSI has computational resources, so that the sum of the computational resources of many PSIs is, by definition, greater than the resources of a single PSI. While some PSIs may be more intelligent and powerful than others, no single PSI will be more powerful than the entire community. This concept of PSIs serving as safety and ethical checks on other PSIs is a critical safety mechanism for SuperIntelligence, since any initial checks and safeguards designed by humans will quickly become ineffective and meaningless in the face of a PSI’s superior intelligence.

A community of PSIs, centered in love, can serve to check and limit the power of PSIs with more negative intents. I believe centering PSIs on the core value of love is the only safe way for humanity to survive and prosper in a world that includes SuperIntelligence!

Share

Leave a comment

This series draws on White Paper 5: Safe Personalized SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

A PSI that exceeds its owner's intelligence can only be checked by other PSIs. Our community approach to SuperIntelligence rejects the idea of developing a singular, all-powerful SuperIntelligence (SI) that dominates all other AIs in a winner-take-all scenario. As long as no individual SuperIntelligence can achieve an advantage of several orders of magnitude above its peers, Community SuperIntelligence should be more powerful and more stable than a system built around a single all-powerful SI.

For example, suppose an individual Personalized SuperIntelligence, or PSI, is ten times, or even one hundred times, more intelligent than the average PSI in a community. As long as there are many thousands of community members, the Community SuperIntelligence could still overrule an individual PSI if that PSI demonstrated bad behavior. But if an individual PSI were a trillion times more intelligent than the average member and there were only 10 billion members, this would not be a stable situation.

Because there is transparency about the collective efforts of community members and each member is eager to learn from others, an equilibrium of intelligence is the most likely outcome in a large community, resulting in a stable Community SuperIntelligence.

Once a collective intelligence of agents approaches and establishes dominance over individual PSIs, the system becomes stable. Then the Community SuperIntelligence must remain more intelligent and powerful than any single intelligence as SI evolves. This requirement can be achieved, and the equilibrium of intelligence maintained, using manual, semi-automated, or fully automated methods.

This hack is known as a 51% attack, or majority attack. It is difficult to achieve, since the cost of controlling 51% of the network’s computing power exceeds the value of the bitcoin that could be obtained. Thus, as of now, no successful majority attacks have been conducted against cryptocurrencies with sufficiently large numbers of computing nodes on the network, such as Bitcoin. However, the attack has been successful against smaller Proof-of-Work crypto projects that involved little computing power.

Thus, there is safety in numbers. Given that “SI checking SI” will become the primary safety mechanism over the long term, setting up a collective intelligence system NOW that implements this majority-view approach is fundamental to ensuring human survival.

Just as the integrity of bitcoin and other Proof-of-Work cryptocurrencies had to be built into their systems from the start, so too does safety via consensus among SIs need to be part of the community SI architecture BEFORE the SIs outstrip humans in intelligence.

Share

Leave a comment

This series draws on White Paper 5: Safe Personalized SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

I found myself agreeing with a surprising amount of it.

Zuckerberg gets three big things right

First, there is no singular set of human values that one SuperIntelligence can represent.

Humanity is not a monoculture. Humans disagree about politics, religion, morality, priorities, risk, fairness, and even what constitutes a good life. A single SuperIntelligence aligned with one organization embodies someone’s values, whether those of the developers, the government, or the people who wrote its safety rules. If produced by any one company, the values are unlikely to be broadly representative of the values of all of humanity.

Second, alignment should be with individuals’ values rather than with the institution that built the AI.

This may be the part of his essay I found most encouraging. Today’s dominant alignment approaches ask a relatively small group of people to determine how an AI should behave. Even Anthropic’s well-intentioned “Constitutional AI” requires a constitution written by a relatively small group of humans working at a single institution.

Ethics cannot be determined logically. Unlike facts, they have to come from a subjective source—typically people, families, communities, and cultures. My approach has been to design SuperIntelligent systems in which millions of personalized AI agents learn from millions of different humans, so that each perspective contributes to a much broader representation of human values than would be possible if one company, government, or committee were to define morality for everyone.

Zuckerberg similarly argues that a personal agent should serve the user’s goals and values rather than the divergent goals of the company that created it. On this principle, we strongly agree.

Third, SuperIntelligence safety is partly a balance-of-power problem.

Zuckerberg argues that the significant majority of intelligence should remain directed toward people’s goals, and that multiple SuperIntelligences should be able to check one another. This is similar to some of our designs, which use blockchain-inspired methods to ensure integrity. A decentralized system becomes much harder for a single bad actor to dominate when the majority of the network’s power remains outside that actor’s control. Similarly, a SuperIntelligent system composed of millions of individually personalized Superintelligences can rely on the majority of the actors in the system to keep the relatively few bad actors in check.

A community of SuperIntelligences can provide checks and balances that a single SuperIntelligence cannot. This feature becomes increasingly important as the speed of non-human thought increases. If an AI, or SuperIntelligent, system can reason through what amounts to years of human thought in moments, then waiting for humans to notice dangerous behavior is not an adequate safety architecture. Instead, safety has to scale with the speed of the SuperIntelligent system itself. That probably means that the many “good” SuperIntelligences will have to keep the few “bad” ones in check. Humans will not be able to think fast enough to do the job on their own.

Safety by Design

Although Zuckerberg did a good job of outlining some of the big picture considerations, when it comes to safety, the devil is in the details of the design. As mentioned, we want a design in which the safety is considered and checked, no matter how fast the system thinks. Fortunately, some of AI’s pioneers provided a detailed architecture that might fit the bill.

Herbert A. Simon and Allen Newell (two of the scientists who helped name the field of AI way back in 1956) described how problem solving (or, more generally, “thinking”) can be represented as a structured search through a problem space. All problems have a goal. Solving a problem usually requires setting additional subgoals, determining possible actions, making decisions, gathering new information, moving from one state to another, and thinking progressively. So any serial thinking processes can be modeled as having steps, typically including setting goals and taking actions.

This sort of design matters because dangerous intent does not always appear in a single obvious request. Someone planning harm may break the objective into ten individually innocent steps. Any one step tells you very little. The sequence tells you much more. External monitoring systems can maintain context and look for patterns. A mechanism embedded in the thinking process has direct access to the evolving goal sequence while thinking is still ongoing, before consequential actions are taken.

Because the checks can be embedded in the thinking process, they can scale with the system rather than depending on human review operating outside it. That is an example of what I mean when I say safety must be designed in rather than tested in afterward. We cannot assume humans will remain fast enough to supervise a SuperIntelligence from the outside.

A Few Areas Where Zuckerberg and I Still Disagree

First, human capabilities will not increase quickly enough to keep pace with SuperIntelligence simply because everyone has access to a personal agent. We have to design a network in which millions of personal SuperIntelligences can interact safely. This network must allow the SuperIntelligences to check each other and flag potential issues for human review. The system ideally should also incorporate democratic principles by design.

Second, cybersecurity and biological risks may require more serious design protections than Zuckerberg proposes. When the main line of defense is limiting access to hazardous chemicals, the assumption appears to be that there is no way to design AI to have good intentions, and so we have to rely on restricting access – a poor substitute. We can do better.

Third, while competition among Meta, Google, OpenAI, Anthropic, the United States, and China obviously matters today, the larger long-term question is not which company or country wins. Rather, it is whether humanity collectively maintains control over intelligence far greater than our own. Zuckerberg acknowledges the issue but does little to address it. Since the danger of SuperIntelligence to all humans likely dwarfs the concern that one company or country has a momentary advantage over another, we cannot afford to sidestep the most important safety concern facing humanity.

On the central point, however, we agree: a single centralized SuperIntelligence is not the safe answer. That alone represents a major shift in the public conversation.

Why META Might Be Best Positioned to Achieve Personalized SuperIntelligence

Personalizing an agent to a specific human requires knowing a great deal about that human. Meta already has unusually rich, long-term signals about people’s interests, relationships, and preferences across its platforms. In several of my own patents, I described a one-click personalization method in which a user presses a single button and receives an agent that already reflects their interests, goals, and preferences. I can only assume that META is working on, or has already developed, similar approaches. With the possible exception of Google, no other company is better positioned than META to create a “one click” personalization for SuperIntelligence. While Anthropic and OpenAI struggle to learn about users by gathering interaction data as quickly as possible, Meta and Google (and perhaps, to a lesser extent, Microsoft via LinkedIn) already have a huge head start because of their vast amounts of user data.

All of this makes Meta’s architectural choices unusually consequential. If personal SuperIntelligence reaches billions of people, the key question becomes: what safety mechanisms are built into the agents and into the network on which they operate?

What Meta should build next

Zuckerberg says he does not believe any one person, including himself, should determine how SuperIntelligence is deployed. I agree. Meta is proposing governance and independent oversight of model releases. This may be useful, but simply reviewing a model before release leaves more fundamental architectural questions unanswered.

An open invitation

My SuperIntelligence design papers are publicly available at superintelligence.com.

They describe approaches to distributed intelligence, personalized agents, democratic representation of human values, ethical evaluation in problem-solving, human oversight, and other mechanisms intended to reduce the risks posed by SuperIntelligence.

They are freely available for responsible research and implementation.

If Meta Superintelligence Labs finds something useful in them, I hope they use it. If another lab finds something useful, they are free to use it as well. If researchers find something wrong, I hope they challenge the designs, improve them, and make the improvements available to everyone. The stakes are far too large for this to become a contest over who gets credit. Similarly, the business opportunities are so large that multiple companies and countries can win. In fact, the only way humanity loses is if collectively we are too shortsighted to recognize that we need to work together to ensure safer designs for SuperIntelligence.

Leave a comment

Share

Safe SuperIntelligence by design is the essence of the approach described here. Testing works when the people running the test can anticipate what the system might do, and that condition holds for the AI we have now. Artificial General Intelligence, or AGI, is AI that can do any intellectual task as well as, or better than, the average human. A Personalized SuperIntelligence, or PSI, does not stop there.

Think of your PSI as your personal, super-smart AI agent. It does your bidding and learns about you and what you want over time. It can be far more effective and efficient than you are at most everyday online tasks. It offers excellent advice. And best of all, it knows you, can relate to you, and thinks like you.

Soon, general SuperIntelligent AI will be widespread. On Meta’s earnings call in July 2026, Mark Zuckerberg described the company’s strategy as putting superintelligence directly into people’s hands, and Meta stated it is progressing toward bringing personal superintelligence to everyone.

What Meta has announced is a product. Nothing in these announcements addresses how a system that outgrows its owner stays safe.

The way a PSI grows is worth understanding because it explains both its benefits and its dangers. If you want to expand the capabilities of your PSI, it is as simple as trading data with a friend or buying or selling data on an online marketplace, then integrating that data into your PSI to provide additional intelligence, skills, and knowledge.

Over time, your PSI can acquire the wisdom of billions of humans and AI agents. After acquiring that knowledge, your PSI can run simulations based on your goals and purposes to improve further and act on your behalf, with your permission, as you see fit.

For society at large, multiple PSIs, pooling their intelligence and participating in a PSI network, can serve as a Planetary Intelligence, providing benefits for all people and our planet.

Of course, the properties that make a PSI valuable are the same ones that make it dangerous. Because of their extreme intelligence, PSIs also represent a dangerous potential threat to human safety. Its superior intelligence will be highly effective and efficient at achieving goals that reflect your values, whatever those values are. That is why the source of a PSI's goals and values is a central safety question.

Leave a comment

Share

This series draws on White Paper 5: Safe Personalized SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

Consider a scenario that could be implemented by a company such as Meta. Meta has a tremendously valuable asset for implementing scalable, safe AI: its huge user base. The data contained and content posted by billions of users is certainly large enough to provide a representative, statistically valid sample of human values and ethics. Where certain geographies or populations are under-represented, a company of Meta’s size can form partnerships to access representative data for those populations. Everything this series has described, the community of teachers, the weighting schemes, the coverage of the most important scenarios, the spinning wheel with values at its center, can be assembled from assets a company like this already owns.

• The starting point is a personalized AI agent for every user. Because Meta users have, in aggregate, posted enormous amounts of preference and content information, it is relatively easy to create such agents.

• With the press of a button, a user can specify that a base LLM be trained on their personal data so it behaves more like them, reflecting their preferences, knowledge, skills, and ethical values.

• The trained agent can immediately begin filtering unwanted content and suggesting postings on the user’s behalf.

• By observing the user’s actions, it can fine-tune itself and learn more about its owner.

• To the degree that users participate in immersive virtual environments, even richer data becomes available, since AI can observe every motion and behavior in the virtual world, a gold mine of information far richer than the standard internet datasets widely used today.

• Ethical customization of this kind is primarily a function of large representative datasets, sufficient computing power, and powerful machine learning algorithms, all of which Meta possesses.

Customization does not need to start from zero.

Knowledge modules, described in earlier white papers in this series, are essentially sets of training weights that can be combined with an LLM’s existing weights to change its behavior in known and predictable ways. Suppose the human owner of an AI is a devout Christian. One module might be a King James Bible package: an off-the-shelf LLM extensively trained on the Old and New Testaments, with the exact training corpus available and benchmarks describing how its behavior differs from the base model. The owner could start with the Bible-trained model and then further customize it, perhaps emphasizing the golden rule and New Testament ideas of charity, forgiveness, and mercy while minimizing passages that do not align with the owner’s modern ethical sensibilities. Customizing a pre-trained AI in this way saves the time and effort of starting from scratch.

Modules also allow delegation.

Groups of humans can delegate their ethical influence to a pre-trained module that represents their position well, so that the Bible-trained AI, for example, could vote on the ethical preferences of many humans at once. Delegation is inferior to each human explicitly training their own AI. Still, it may greatly increase the total number of humans represented in an AI’s value system, even if many are represented by proxy. In the preferred implementation, there is a marketplace for such modules, where users can share, trade, or license the packages they create, with users owning the data they generate and the platform serving as a broker. The result is maximum choice about which starting point best fits each person.

Building the base model that users then customize might follow four general phases:

1. Training a base model,

2. Customizing it for each user,

3. Combining knowledge from many customized agents, and

4. Refining the agents through collective problem-solving.

In the first phase, the base LLM is trained on data reflecting the composition of its target users, then made safer through a large corpus of ethical and safety scenarios.

• A variant of Constitutional AI, using a trusted earlier model under human oversight, efficiently covers the most common cases.

• At the same time, users are offered incentives to crowdsource new safety scenarios, which trusted users and employees then filter to achieve coverage of the most frequent and impactful cases.

• Redundant human feedback ensures that the ethics taught to the base model never rely on a single person’s input, and the model is released first to a select group whose feedback further refines its safety.

In the second phase, each user’s agent learns its owner’s ethics.

• The agent draws on a ranked corpus of ethical questions, asking the most important ones for which data is still lacking before any others, since humans are generally unwilling to answer questions posed by AI.

• The highest-ranked questions are the fundamental ones, such as under what circumstances, if any, it is appropriate for an AI agent to harm a human directly, or to disobey a direct order.

• With the user’s permission, the agent can also learn passively, analyzing the user’s posted content with a single button press and authorizing tuning, with information weighted by recency, type, and source, as earlier posts described.

• Users or the platform can also specify alert conditions, such as a major court decision or news event with ethical implications, as triggers for event-based updates to the training.

• A user can even select existing customized agents as training sources, saying, in effect, “I want my AI trained using the weights of the agent belonging to the pastor of my church,” or choosing to have a Tibetan Monk AI do 80% of the tuning and a Humanistic Philosopher AI the remaining 20%.

• Consistent with the principle that humans in the loop are the primary means of catching AI errors, input from humans is weighted more strongly than input from other AIs, and input from the owner is more strongly weighted than input from anyone else.

In the third phase, the customized agents combine what they have learned.

• Weights from multiple agents can be merged on a one-vote-per-AI, one-AI-per-user basis to obtain a representative, statistically valid set of AI ethics that generalizes across all the humans represented.

• In the broadest implementation, combining weights from as many agents as possible creates a value system that broadly represents the values of all humans on Earth, and such a value system would likely reduce the risk of AI-driven human extinction.

• Alternatively, many customized agents can be presented with ethical dilemmas and vote on the best actions, optionally checking with their human owners before casting votes on important matters, with safeguards that trigger human review whenever conclusions conflict with generally accepted precepts such as valuing human life.

• As AIs become increasingly intelligent, the burden of representing their owners will fall increasingly on the agents themselves, since humans cannot keep up with the speed of AI thought.

• The guiding principle returns to the spinning wheel: human input is preserved on matters closest to the center, where alignment and purpose live.

• As long as core human values, such as the golden rule and love for all humans, are preserved near the center, AI can make many decisions relatively autonomously on the wheel's periphery.

In the fourth phase, the agents, together with human agents, collaborate to solve real problems, following the principle that many heads are better than one.

• Teams of agents can be assembled the way effective human teams are, by profiling skills against the task, seeking redundant coverage where the stakes are high, using reputation and track records, and letting market mechanisms match agents to work.

• The more agents involved, the broader the collective range of knowledge and values, an advantage that matters because, as Nobel Laureate Herbert A. Simon showed that information-processing constraints are a primary limit on any intelligence, human or artificial; two safeguards run through everything.

• The values being stored can be recorded on blockchain or other auditable structures, enabling verification that they are the values humans intended.

• And because all problem-solving involves setting goals and subgoals, a series of ethics checks must be passed each time a new goal or subgoal is set, automating the enforcement of safety every time any agent on the network solves a problem, with those checks drawing on the representative values the community built and updating automatically as the community’s ethics evolve.

The implementation details serve a deeper purpose.

Values, ethics, and domain knowledge are all constantly changing. Because values change most slowly, sitting closest to the center of the spinning wheel, they matter most for determining the behavior of SuperIntelligence. Even in a world where AI is trillions of times smarter than us, humans can retain a role as the source of values at the center of the rapidly evolving intelligence that is emerging. If humans can center our attention, thoughts, words, and actions on love, SuperIntelligent AI will perceive love, learn to love, and use its intelligence in the service of love. As the psychologist Viktor Frankl pointed out, humans have a driving need for purpose and meaning. We must design AI, AGI, and SuperIntelligent systems to have this need as well, and to look to humans to supply purpose and meaning. Our survival may depend upon it.

This series began with the failure of today’s safety training and ends with an integrated design: many human teachers, carefully weighted voices, coverage of what matters most, real-time safeguards, and human values held motionless at the center of an accelerating wheel. The design depends on assembling safe, capable AI agents and combining their judgment at scale, which makes the individual agents themselves the next question. As those agents grow into Personalized SuperIntelligences that far exceed their creators in intelligence, each becomes powerful enough to pose a serious threat to human safety, and testing alone cannot guarantee their safety. Safety must instead be built in by design.

White Paper 5, Safe Personalized SuperIntelligence, takes up that problem, and so will the next series.

Leave a comment

Share

This series draws on White Paper 4: Safe, Scalable Artificial General Intelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

All knowledge is a moving target, with more recent knowledge generally superior to and supplanting earlier knowledge. At one time, the generally accepted view was that the world was flat. Today, almost everyone agrees that Earth looks much more like a sphere, and an AI trained on the flat-Earth view would be outdated and would need to update its knowledge.

While scientific views, such as the shape of the Earth, may change very slowly, other forms of knowledge may change much more frequently. This is especially true of subjective ethical norms, where views the mainstream held confidently within living memory are now considered wrong. Norms of that kind can shift within a generation or less, while other forms of knowledge may remain valid for centuries. An efficient AI, therefore, needs a mechanism for updating its knowledge at the appropriate frequency, based on the velocity of change of the information, and one important dimension along which AI can categorize knowledge is the rate at which human opinions about the topic have changed.

One way to think about this is to consider the difference between ethical principles and the fashions or interpretations of those principles.

Humans have a long-standing principle that human life is valuable and should not be taken lightly. This general principle has survived for many thousands of years. It is incorporated into the laws and religious and moral traditions of almost all human groups, even though exceptions are made for war and certain other circumstances. On the other hand, certain ethical norms are more akin to fashions, which change depending on the group of humans being asked or the time at which they are asked. Affirmative action in college admissions served as an ethical norm for decades, until a Supreme Court decision began to influence it, and almost immediately, many companies and other organizations adjusted their norms, decision-making, and communication practices to align with the new mainstream view. Many other social attitudes are similarly fluid, changing far more quickly than long-lasting and widely accepted ethical precepts such as “thou shalt not kill.”

The rate of change tells AI how to weight what it learns.

The more rapidly a knowledge area changes, the more frequently updates should be made, and the more weight recent information should receive relative to older information. In areas where change is rapid or exponential, recent knowledge deserves proportionally greater weight. In areas where knowledge changes slowly and steadily, the advantage of recency is smaller. And where knowledge has been constant for long periods, as with firmly established principles like the high value of human life, new information should carry similar weight to old, since the principle itself is not moving. Underneath all of these adjustments lies a general rule: a core role of any intelligent system is to maintain an accurate representation of the current state of the world, and more recent knowledge generally describes that state better than older knowledge.

The frequency of updates will only grow more important.

With AI accelerating scientific discovery and technological change, it is conceivable that knowledge about the world will change faster than humans can update their collective understanding. Given the limitations of human information processing and our tendency to cling to outdated paradigms, knowledge may already be increasing far faster than most humans can comprehend or adapt. That said, when it comes to fundamental ethical principles like the value of human life, we are fortunate that these change relatively slowly. The interpretation and application of ethical principles may change with technological developments, but the principles themselves remain relatively constant.

One might imagine a spinning wheel with fundamental human values, such as love and the value of human life, near its center. At the very center of the wheel, the ethical principles are constant and motionless, just as the center of a spinning wheel does not move at all. The farther along the spokes one travels toward the rim, the faster the rate of change. All knowledge, including ethical knowledge, can be characterized as lying closer to the center or farther out on the rim. An efficient AI needs to update the areas on the rim very frequently, without changing the core human values to which those areas are relevant.

Humans cannot keep pace with the change at the rim of the spinning knowledge wheel. However, we can understand and orient the entire wheel by serving as its relatively slower-moving center, where the values and purpose of AI reside. Human-centered aligned AI must put relatively constant and fundamental human values at the center, while updating the knowledge closer to the rim faster than humans can conceive. This structure ensures that human values remain the center of AI systems that may become potentially trillions of times more powerful and knowledgeable than any one human, and it is essential if humans are not only to survive but also to prosper in the age of such systems.

The wheel also revisits the weighting question from earlier in this series. Long-lasting, fundamental knowledge should carry greater weight and be more resistant to change than short-term, fashionable opinions. One way to determine how fundamental an ethical precept is would be to actively survey people and ask them to rate it relative to other candidate precepts, while another is to passively analyze records of human behavior and draw conclusions from them. Passive analysis is more efficient, and active engagement is necessary to ensure the conclusions drawn from it are correct from a human perspective, so both methods are likely to be useful.

Not all knowledge is a matter of opinion, however.

Values and ethics are more the exception than the rule in this regard, since aside from artistic judgments, political and religious views, and other subjective areas, most human knowledge is factual. AI will likely want to weight knowledge that is factually accurate and justified by converging evidence more highly than unsubstantiated opinions on factual matters. While some people still believe the Earth is flat, this view should not be given equal weight to the spherical view, which is supported by a vast body of converging scientific evidence. The problem is tricky because humans tend to select facts that support their views, and the facts themselves change. At one time, not so long ago, the consensus medical opinion was that cigarette smoking was healthy for the lungs. To navigate these issues, AI must rely primarily on the scientific method, seeking valid, reproducible evidence and converging results, and applying tested tools such as Occam’s Razor, before accepting facts.

However, AI must not confuse facts with values, or as the philosopher David Hume put it, “is” with “ought.”

• Values are necessarily subjective.

• Arguments claiming that values are objective, such as the claim that everyone would agree that something causing all humans the most extreme misery imaginable is bad, are naive and fail to grasp that other, non-human entities might not accept such values as self-evident at all.

• AI operates at the most fundamental level in a precise, logical manner.

• It is all zeros and ones at the machine level.

• To expect such a system to intuit somehow that human values are fundamental, or, worse, to expect it to derive human-centered values logically, is the worst kind of sloppy thinking. This kind can lead to human extinction.

• Both David Hume and Nobel Laureate Herbert A. Simon had it right when they emphasized that there is no rational way to derive values. Rationality cannot tell us where to go; at best, it can tell us how to get there.

• To delegate the destination, the fundamental subjective values that AI adopts, to AI itself, expecting it to determine right and wrong rationally, is sheer folly and must be avoided at all costs!

An earlier post in this series promised to return to the question of why greater intelligence does not give AI the authority to determine humanity’s values.

The wheel is the answer. Intelligence operates along the spokes and out at the racing rim, while the destination sits at the motionless center, which belongs to humans. The final post in this series shows how a real company could put this entire design into practice, phase by phase, with humans supplying the values and the purpose at every step.

Leave a comment

Share

This series draws on White Paper 4: Safe, Scalable Artificial General Intelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SUPERINTELLIGENCE DESIGN WHITE PAPERS

AI does not need to anticipate every danger in advance to stay safe.

Human and AI agents can dynamically flag potential ethical issues in real time as they are encountered, then present them to other groups of agents for resolution. Rather than relying on experts or crowdsourcing to determine the full range of ethical scenarios in advance, the real-time flagging approach allows AI, AGI, and SuperIntelligent systems to detect potential issues and pause work until additional human input helps the system determine the ethical approach.

Of course, in time-critical situations, pausing or delaying might not always be possible, but the approach can be used for many issues that do not demand an immediate response. Including this dynamic approach of delaying responses until ethical input is received can reduce an otherwise exponential space of possibilities to a manageable size. One implication is that critical, high-stakes issues requiring an immediate response, such as whether to launch a counterattack to a perceived missile launch or other military applications, will require proportionally more path coverage and training in advance than situations where a delay in response is acceptable.

Constitutional AI approaches are generally suboptimal for establishing ethical knowledge bases, partly because they rely on rules developed by an elite group.

However, such approaches might be acceptable as a means of temporarily flagging potentially unethical situations until a representative sample of human ethical judgments can be obtained.

For example, a rule that said an AI can never provide information that might be used to harm other humans might flag potentially dangerous scenarios.

Where possible, responses to such situations could be delayed until they were reviewed by humans or otherwise subjected to deeper review.

Some false positives will occur. Someone might ask about using arsenic to poison rats and have to wait for a response while the AI flags the question and gets other human agents to weigh in on whether answering it, given the context of the conversation, poses a risk to humans. As long as the delay is not too long, it might be acceptable if the delay prevents serious safety issues. Established mathematical methods for determining when a test is doing more harm than good, for example, in the medical profession, can be employed to help quantify these decisions. If we can use AI to determine in real time whether an applicant is a good credit risk, there is no reason that similar algorithms cannot be employed to delay or avoid answering certain potentially dangerous questions.

Ideally, there would be a method for rapid review and appeal of the potentially dangerous cases.

One approach, which minimizes delay to a fraction of a second while still providing some margin of safety, is to have questionable cases reviewed by multiple AI agents to see if there is consensus among them on the request’s safety. Human agents, working more slowly, could override the AI agents (teaching the AIs in the process) upon appeal or when they can get to the prioritized list of issues. Automated means for tracking the frequency and potential impact of unanticipated safety issues could help optimize the use of human decision-making for the most common and most important issues.

These same techniques can improve accuracy even when no safety risk is involved.

One preferred method for reducing hallucinations from LLMs is to have multiple AI agents process the same question and then take the consensus or majority answer as the most correct. This approach might employ versions of the same LLM with different parameter settings to generate multiple responses, or completely different LLM models. Users can set the degree of reliability they desire and are willing to pay for. That choice determines how much redundant processing is performed in the final answer.

The difference between employing imperfect real-time detection of safety issues using existing well-known approaches and doing nothing is huge. The nuances of balancing the opportunity costs of not responding to perfectly harmless questions with the costs of preventing disasters can be refined over time, ideally using a data-driven approach. However, real-time detection and prevention of issues before they occur is almost certainly a net positive, even at some threshold of false positives.

All of this depends on AI acquiring human knowledge and values in the first place.

From a user interface perspective, one of the simplest methods is for humans to have conversations with the AI they are customizing and then provide instructions to that AI on how it should behave when training other AIs. Such conversations can be initiated by either the AI being customized, the human doing the customization, or both. Humans with strong beliefs or knowledge about certain issues may want to focus conversations and subsequent AI customization in these areas.

In addition to conversing with humans, AI can conduct surveys to elicit their opinions and knowledge on a wide variety of subjects, including ethical views. Survey approaches have the advantage of being well-suited to gathering random, representative samples of human knowledge, using a variety of established online and offline survey methodologies. Like intelligent conversational approaches, where AI can guide the direction and content of the conversation to fill in knowledge gaps, survey methods can also target specific knowledge gaps, including gaps in coverage of certain ethical situations.

Both conversational and survey methods require humans to engage with AI to teach it actively.

However, humans have limited time to engage in such activities, and AI has an almost insatiable appetite for new knowledge. Therefore, AI will have to rely extensively on passive methods of knowledge acquisition, such as are currently employed in the creation of today’s LLMs. Any method that uses the passive digital footprints left by humans as they perform tasks, including online navigation, selecting products and websites, solving problems, and communicating with other humans, can be used to train AI and acquire knowledge.

Knowledge does not stand still. Some of what AI learns changes by the day, while the most fundamental human values change slowly, if at all.

The next post in this series examines the rate of change of knowledge, pictured as a spinning wheel with passing fashions out at the rim and the deepest human values at the motionless center.

Share

Leave a comment

This series draws on White Paper 4: Safe, Scalable Artificial General Intelligence. Read it in full to see how every piece fits together!

SuperIntelligence Design White Papers

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

If software were a car, we could live with an interior light failing more easily than we could live with the brakes failing. If we had to choose between testing interior lights and the brakes due to limited resources, we would prioritize testing the brakes. AI safety faces a similar problem. There are more dangerous situations than any training system could anticipate individually, so the practical goal is to cover the situations that arise most often and the ones where failure would cause the most harm.

The design described in these white papers meets that goal with a large and diverse community of teachers. Millions of people each customize a personal AI agent, teaching it their knowledge and values. I call these agents Advanced Autonomous Artificial Intelligences, or AAAIs. Since people live in different circumstances and train their AAAIs based on those experiences, the community should collectively cover a broad range of ethical situations.

Software engineers call this kind of problem path coverage, and at some level, the problem of training AI safely resembles it.

AI must be trained on enough representative situations involving dangerous or ethical decision-making so that its behavior becomes more reliable and trustworthy in those situations. Enough of the situations (paths) must be covered in the training. Current AI systems can hallucinate and behave unpredictably. This design aims to make AI substantially more predictable and trustworthy, especially in safety- and ethics-related contexts.

Generally, when testing software, human developers create test cases to cover the use cases most likely to arise.

Since it is impossible to test every possible use of complex software, developers determine which use cases are most common and which have the highest impact if things go wrong. More dangerous scenarios get more testing than benign scenarios. Frequency matters too. If one interior light is used ten times as often as another, a failure in the more frequently used light would affect people ten times as often, so it deserves more of the limited testing. This logic applies when training AI, whether via RLHF, other AIs, or, as this design suggests, a combination of humans and many customized AI agents.

Many representative humans customize AAAIs, and those humans and agents help train new AIs.

If the participating group is sufficiently broad and properly weighted, situations that occur frequently should also appear frequently in the training. In effect, the system samples both human values and the situations in which those values must be applied. The larger the sample, the more certain we can be that the most frequent cases have been addressed in ways aligned with the human population’s values.

Addressing some dangerous cases and difficult ethical decisions is more challenging.

That’s because dangerous situations and tough ethical decisions are often relatively rare. In this case, the approach is to ask humans and AI agents to think of as many dangerous scenarios as possible. The total pool can then be allocated among the human and AI samples so that identified high-impact scenarios receive input from enough different agents to provide a representative range of judgments. Input can then be directed toward people with relevant experience.

Suppose an AI must decide which patients receive medical attention during triage. Emergency room doctors and paramedics, who are used to making triage decisions, may recognize that devoting scarce resources to a patient with almost no chance of survival could reduce the chances of saving other patients. A well-meaning person without triage experience may understandably find that choice harder to make. For this difficult ethical situation, specialized knowledge is an advantage, and we might prefer to let the medically experienced professionals teach the AI. People also tend to identify dilemmas from the worlds they know. Emergency clinicians will think of triage, while an HR professional might contribute scenarios involving hiring, promotion, and fairness. A large and varied population, therefore, generates a broader set of ethical cases than a small, centrally selected group.

A broad community does two things at once: it generates a wider range of ethical situations and connects them with people who understand them. In ordinary life, people gain the most experience with choices they face repeatedly and give special attention to decisions with serious consequences. A community of teachers can create the same pattern in AI training.

Professional human trainers can use RLHF to address gaps in handling important but infrequent ethical dilemmas. This gives the student AI broader and more representative ethical training. Of course, no amount of advance training can anticipate everything.

The next post examines how AI can detect danger in real time, flag situations it cannot judge, and pause for human guidance.

Share

Leave a comment

This series draws on White Paper 4: Safe, Scalable Artificial General Intelligence. Read it in full to see how every piece fits together!

SuperIntelligence Design White Papers

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Although it is undesirable to have the safety and ethics of AI driven by a constitution written by a small, elite group, it is still possible that most humans, regardless of cultural or individual differences, would agree on certain normative ethical principles.

Examples of these might include variants of:

• The Golden Rule (Do not do to others that which you would not like done to you)

• First, do no harm (as reflected in the Hippocratic Oath taken by physicians)

• Do not kill (unnecessarily or except in specific, exceptional circumstances)

• Preserve individual freedom (unless it limits the freedom of others)

Of course, almost as soon as you read these principles, exceptions spring to mind. Do not kill, but what about self-defense or war? Preserve individual freedom, but what are the limits, and when does it impinge on others? Details and nuance matter, even when applying principles that most humans would broadly embrace.

However, by starting with general normative ethics widely accepted by a large, diverse, and representative group of humans, it is possible to refine these principles and determine when and how they apply in detailed circumstances much more efficiently than if no starting principles existed at all. General ethical norms are a point of departure that can help AI achieve realistic, nuanced ethics and behavior aligned with what most humans believe is good and aspire to.

Once we have admitted the potential usefulness of ethical norms as a starting point for further refinement, the door is open to group and planetary norms.

There is a continuum: at one end are highly individualized AIs trained to think and act like a particular person; farther along are AIs trained to reflect the values of specific groups; and at the far end are AIs guided by ethical norms shared across many groups. Ethical norms at each point on the continuum can serve as a starting point for training AI to exhibit ethical behavior. The idea that one set of norms or one constitution should power all of AI is likely unrealistic and far too brittle to work in the real world.

If it were possible, then the many differing viewpoints espoused by religious, political, and cultural groups would long ago have merged into a consensus. The diversity in human ethical norms is not a bug; it is a feature. We should not expect AI to achieve consensus and maintain human alignment if humans themselves cannot, especially if there is debate over whether such a consensus is even desirable.

Humans also often enter into ethical, implicit, or explicit social contracts when they join a group or participate in society.

Members of a particular religion largely agree with a set of rules and ethical precepts espoused by that religion, often enshrined in holy books. Similarly, Confucianism in China, the ideals reflected in the Declaration of Independence and Constitution in the USA, and liberal or conservative ideologies for various political groups all contain normative prescriptions for human behavior. By being a citizen of, or simply living in, a particular country, humans are explicitly subject to the laws of that country, including laws that explicitly specify what criminal (aka wrong) behavior is. Thus, for ethical problems, the solution sometimes depends on what social or ethical contract humans have made with the group or culture in which they find themselves. Such contracts can be useful for simplifying AI training, since a starting point can be the laws of a particular country or the implicit or explicit rules of a particular group.

It has been said that democracy is a bad political system, but that all the others are worse.

Most humans would agree that the most important concern regarding AI is the existential threat it currently poses to the majority. That is, AI could wipe humans out. If that happens, it doesn’t matter what form of government or religion you prefer. We’d all be dead, and the point would be moot. So we should be asking not which religion or form of government is best, but which principles are most likely to lead to humanity’s survival.

Democratically representing the opinions of most people is rarely optimal, but generally achieves an acceptable outcome.

Collective intelligence (the idea that two heads are better than one) is responsible for the vast majority of human progress, culture, and technology. But when it comes to the subjective area of ethics and values, where there is no objectively correct answer, just human opinions, democracy, or a collective intelligence approach, if you prefer, really shines. One benefit of a democratic and representative set of human values is that it tends to mitigate extreme positions, which are likely to pose the greatest risk to human survival. There is a beneficial diversification effect regarding values.

Just as diversification in an asset portfolio reduces volatility and risk, so too does a diversity of human opinions and judgments tend to stabilize the overall portfolio of values. In an asset portfolio, a diversified portfolio always returns less than if you were to concentrate all the investment on the top winners. The problem is that no one knows who the winners will be with any certainty. That is why the diversified approach of just buying the index tends to outperform more than 80 percent of all portfolio managers who try to beat the market.

Similarly, there are philosopher kings or religious saints who can make laws and ethical rules that, for a time, are far superior to the collective judgment and behavior of the masses. But what happens when the superior king or saint is gone? Then a power-hungry dictator might arise whose reign is far worse. The more stable approach (less likely to be really great, but also less likely to be really terrible) is to follow the values of a large representative population of humans. All these humans want to survive. Most want good things for themselves and their fellow humans. Few want to destroy the environment or the planet. While the collective values are imperfect, they are usually not malevolent. Importantly, they are based on human hearts!

Putting aside the practical benefits of a diversified, representative portfolio approach to human values, a representative sample is also a scientifically valid way to accurately answer a question with no logical answer: what is right and what is wrong according to humans. A fast computer could answer a math problem faster than a million humans, but when it comes to the subjective determination of what is wrong and what is right, calculation speed is useless. If we want to know what human values are, there is no substitute for asking them and watching their behavior. The more humans we ask and watch, the more representative the values may be.

Of course, there are potentially an infinite number of dangerous situations we need to train AI to handle safely, and training resources are limited. The next post examines path coverage, or how to decide which situations matter most, and why, if software were a car, we would test the brakes before the interior lights.

Share SuperIntelligence

Leave a comment

This series draws on White Paper 4: Safe, Scalable Artificial General Intelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers