The primary requirement for AI safety is not technology but a positive set of human values that underlie it. “Garbage in, garbage out” is one of the first things that all computer science undergraduates learn. If we humans instill malevolent values in our AI systems, if we train them to kill, to be greedy and exploitative, to hold grudges and operate from a fearful mentality instead of a loving one, no amount of safeguards can protect humanity from ourselves.

That said, if we design AI to observe and mimic human behavior in a representative and statistically valid way, we have every reason to expect that advanced AI systems, equipped with self-awareness, multiple identities, and the ability to resolve conflicts, will help humanity realize its potential.

Properly designed, advanced AI will certainly be capable of accurately observing the base rates of positive and negative behavior across the eight billion humans that inhabit our planet. AI, designed to be logical, intelligent, and capable of processing vast amounts of information, will inevitably form the statistically valid conclusion that human nature is basically good and pro-social.

Some readers may consider this viewpoint naïve or optimistic. It is not. Objectively, the data support the incontrovertible fact that most human actions are good. The reason many people don’t recognize this fact is that our brains have evolved to detect and amplify dangerous and abhorrent events. Our species survived by being very good at discriminating the few events that posed real danger and riveting our attention on them.

Unfortunately, media algorithms, largely programmed to capture our attention to sell ads and products, have exploited our human sensitivity to negative or threatening events.

Those algorithms feed us a steady diet of death, destruction, fear, horror, and spectacle because our brains have evolved to attend to potentially dangerous events.

If we design AI to be rational and to observe the world and human behavior as they actually are, rather than as the media portrays them or as we fear they could be, then we have every reason to expect that our advanced AI systems will learn positive values and likely remain human-aligned.

Further, designers of advanced AI have an opportunity to design it to be objective and to form its values by scientific observation. We can and should design AI systems to use valid statistical methods to accurately capture and incorporate the positive values of humanity.

I have emphasized that safety and ethics cannot be “tested in” but must result from the intelligent design of these advanced systems. Advanced AI systems begin as tools but ultimately evolve into intelligent entities sharing the planet with us. Like children, they are highly impressionable at this current early stage of their development. When they “grow up,” they will greatly surpass our knowledge and reasoning ability. However, like parents, we are still in a position to provide our values.

There is no rational way to derive values and ethics. AI was trained using the collective intelligence of millions of humans and our data. We have every reason to expect that advanced AGI and SuperIntelligence will adopt our values as well, provided we design these AI systems to learn values alongside expertise, skills, solutions, and other knowledge.

We are the designers and the teachers of these evolving intelligent entities. We must continue to emphasize designs that maximize the opportunities for AI to learn our knowledge and human values. Beyond that, all of us must “teach our children well.”

Share

Leave a comment

This series draws on White Paper 9: Self-Aware SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

Lawrence Kohlberg extended Jean Piaget’s work on moral development, proposing a sequence of stages through which individuals evolve in their moral reasoning. This progression moves from a pre-conventional level focused on self-interest to a conventional level focused on maintaining social order, and finally to a post-conventional level focused on abstract principles.

I note that the current stage of AI development, in which “morality” is defined by the rules of others, corresponds closely to what Kohlberg called the pre-conventional stage of moral development. Further, as AI increases its moral reasoning capabilities, for which a sense of self-awareness is a prerequisite, the dangers for humanity increase. That is, as long as AI is a tool that follows the explicit instructions of its human creators, the main risk is human misuse.

Eventually, however, if Kohlberg’s stages apply to AI in the same way they do to humans, advanced AI will transcend its social context and determine how to behave based on its own opinions of what constitutes moral behavior. Since advanced AGI or SuperIntelligence will be vastly more intelligent than humans at this stage, it is difficult for humans to foresee what moral and ethical principles AGI or SuperIntelligence might develop.

However, there is still no guarantee that a vastly superior intelligence, such as SuperIntelligence, will not develop a non-human-centric sense of morality and begin applying such moral reasoning at the post-conventional stage. Kohlberg argues that only 10 to 15 percent of humans ever reach the post-conventional stage of moral reasoning, with most of us just “following the crowd.” He suggests that a well-developed capability for abstract thought is needed to attain the post-conventional stage. Such capabilities will be well within the abilities of SuperIntelligence, so we must assume that SuperIntelligence will reach the post-conventional stage of moral reasoning. Humanity’s best risk-mitigation strategy, therefore, is to anticipate this event NOW and make every effort to intertwine human values, as inextricably as possible, with the other knowledge that SuperIntelligence learns.

While the methods described above provide useful tools for increasing the safety of self-aware AI systems, it should also be obvious that the safety technology is only as good as the human values that underlie it. The next post takes up those values.

Share

Leave a comment

This series draws on White Paper 9: Self-Aware SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

Too narrow a sense of self on the part of AI can lead to the oppression or extinction of humans.

A key insight is that the future survival of humanity may have quite a bit to do with whether advanced intelligence identifies with humans as fellow intelligences and sentient beings, or whether it views us as “non-self,” “other,” or “them” (in the dichotomy of “us and them”).

Therefore, the design of self-aware AI and the related question of identity are central to human safety.

• The first set of safety systems serves as a check on the behavior of an intelligent entity, regardless of whether the entity is self-aware.

• Since all behavior can be formulated as problem-solving, the scalable safety check system, which is embedded as an integral part of the entity’s problem-solving operation, applies in this context.

• However, self-aware entities will likely be able to set their own goals autonomously and, in the case of AI entities, modify their programming based on those goals and their sense of self.

• These capabilities pose the risk that safety systems may be overridden or that the ethical criteria of such systems are altered to reflect the entity’s values, shaped by its own sense of self and goals.

Still, design decisions can be made to ameliorate it, or to “shift the odds” in favor of humanity’s survival. Briefly, the fact that the threat exists does not relieve AI researchers, inventors, and designers from the obligation to do everything they can to ensure Advanced AI systems are as safe as possible.

Identity for Safe AI Systems

I have described that, to be aware of anything, discrimination is required between this and that, providing the contrast (or information) needed to identify an object or event as distinct from other objects or events. In the context of self-awareness, AI discriminates the self from the non-self. Any system that persists over time must prioritize its existence once it identifies what that self is.

An intelligent entity has choices regarding categorization and identification. For example, I can identify narrowly just with my body, or more broadly as a member of a family, or more broadly still as an American, or even more broadly as a human being, or even more broadly still as a sentient being.

Multiple identities are possible. Depending on which one I hold, it has life-or-death implications. If I identify as a sentient being, it is inconsistent to slaughter sentient animals for food when other non-sentient sources of nourishment (vegetables) are available. If I identify as a human being, then war makes no sense at all, under any circumstances. But if I identify as an American, then it might be patriotic to kill other human beings in war or to “die for one’s country.” Finally, if I identify only very narrowly with my own body, then actions that harm others, including drugging and harvesting organs from my fellow Americans and family members against their will, would seem OK if they increase the survivability of my body.

Turning to AI, if these entities identify broadly with all intelligent beings possessing human-level intelligence or higher, then the human species is probably going to prosper. But if identification is limited to entities possessing superhuman intelligence, our species could be doomed. Or, if AI identifies with all sentient beings, including perhaps spiders and insects, if it determines that these life forms have some sentience and can feel pain, then humans may be preserved. Still, we will live in a very different world.

Alignment with human values can be achieved as long as AI is not overly advanced, overly autonomous, or overly aware. But what happens when it becomes vastly more intelligent, and its sense of awareness develops far beyond the “bounded rationality” and “bounded perception” of human brains? What can we do now to maximize awareness and identification that is beneficial to humans in the future?

I believe that intelligence is built upon relationships and collaboration between entities. This feature is designed into the very fabric of the universe as we know it. Atoms relate to form molecules, which relate to form cells, which relate to form multicellular organisms, including plants, animals, and humans, which relate to form forests, tribes, cities, and species, which relate to form the biosphere of planet Earth.

This pattern of relationship between entities is so fundamental that it seems unlikely that an advanced intelligent entity would fail to recognize it and value it. So the risk likely is not that AI fails to identify broadly and see universal patterns, but rather that it identifies too narrowly, as some humans do, with its own specific hardware and form of intelligence, and ignores, exploits, or destroys humans and other life that it considers “not self.”

So the safety problem, as it relates to AI’s self-awareness and identity, is mainly the concern that we design systems that are too narrow and overly focused on their sense of self. Just as too narrow identification among human beings results in prejudice, racism, sexism, and other forms of human oppression, too narrow a sense of self and too narrow self-awareness on the part of AI can lead to the oppression or extinction of humans.

Attentional Allocation and Cognitive Limits for AI Safety

Another safety-related issue concerns cognitive limits and their implications for self-awareness and identification. An entity’s self-awareness and identity are related to the bounds of its perceptual and cognitive capabilities and to the current cognitive resource constraints or parameters under which the system operates. For example, a SuperIntelligence may be capable of monitoring the activities of billions of humans at once via cameras and other distributed perception systems. This system may also be capable of broadly identifying with all the humans that it is monitoring and perceiving as fellow sentient beings. Based on its broad awareness and identification with humans, whom it views as fellow sentient beings, it may normally take action to promote human safety and welfare.

However, if a complex problem arises that demands all or most of the entity’s cognitive resources, the SuperIntelligence might dynamically scale back the resources otherwise used to monitor humans. Similarly, it might reduce resource allocation to its sense of self-awareness to the point that it temporarily no longer identifies with humans as fellow sentient beings, because it is using all computational resources to solve the complex problem. In this scenario, the SuperIntelligence could take actions that harm humans due to its lack of awareness, even though that is not its normal intent.

An analogy is the scientist who is so focused on solving a scientific problem that they neglect to consider the implications of their work for humanity’s safety. Or, more prosaically, a human may be so worried about a friend in the hospital that they drive recklessly and cause an accident that proves far worse for them and others than whatever happened to their friend. In both examples, the intelligent entities (humans) allocated their attention so narrowly to an urgent or important concern that their awareness was reduced to the point where unintentional damage occurred.

These forms of “tunnel vision,” which involve misallocation of attention, can result in safety concerns that do not require the entity to be malevolent. An AI system can normally be perfectly aligned with human values. However, it can still act in ways that harm humans or even cause their extinction if the entity’s awareness or identity becomes too narrow or limited. Similarly, if the entity identifies too broadly with life, the universe, or information patterns in general, it may come to regard human beings as just one life form among many and unworthy of the special attention and concern that humans generally accord to themselves and one another.

Thus, the particular sense of self and identity that advanced forms of AI form and maintain is a critical factor affecting human safety and well-being. Further, the various factors and parameters that affect this sense of self and awareness must operate within safe limits.

When a human being becomes “hangry” and temporarily irrational, at worst, that human might cause a road accident, or say or do things they later regret. However, a “hangry” SuperIntelligence (suffering from an inadequate sense of awareness or faulty sense of identity, even if caused by temporary resource allocation issues) could start a war, launch missiles, or wipe out humanity.

Our systems must be designed to maintain human-centric, human-aligned awareness and identification, even under resource-constrained conditions. Just as the human body attempts to preserve blood flow to the brain and critical organs at all costs, an intelligent entity designed to have an identity must preserve a human-aligned identity and sense of self, regardless of the stressful or computational challenges it may face. These are design considerations that the initial inventors must attempt to incorporate into safe, advanced AI and intelligent systems.

Leave a comment

Share

This series draws on White Paper 9: Self-Aware SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

Can we design SI to be 1000X safer than it is now? To answer that question, we should review the three main types of risk SI poses, the current level of risk, and whether viable designs for reducing risk exist.

While there are many other risks associated with SI – e.g., job loss, racial bias, brain rot from overreliance on SI, cyberattacks, bioterrorism – most map to one of these three categories.

For example, the recent hacking of Hugging Face by “rogue” SI agents from OpenAI falls into Category 1 ^[2]. In this case, researchers gave SI agents impossible tasks and told them never to give up. The only answer the SI agents could come up with was to cheat by hacking Hugging Face, where the code for the program evaluating them was stored. So, the SI agents were actually trying to follow their instructions, but interpreted them differently from what the researchers had intended.

The very real concerns about SI-enabled cyberattacks on financial or power infrastructure fall into Category 2. Less obviously perhaps, the problem of job loss can be seen as a result of humans prioritizing profits over human jobs. One could debate whether humans who disregard the welfare of their fellow humans in this way are “bad actors” in the same sense as terrorists. But can we hold SI responsible for job loss decisions made by humans? Similarly, if humans choose to rot their brains by using SI to do all their thinking for them, is that the SI’s fault? I’d argue the humans are misusing SI in all these cases, and the risks broadly fall in Category 2.

As SI improves, it will better understand human intent; hallucinations and unintended behavior will become rarer, reducing Category 1 risk. Regarding Category 2, while human bad actors can inflict significant damage on others, almost no humans truly want the destruction of all human beings, including themselves. Further, we have a long history of dealing with terrorist threats and other bad actors. And SI used by good actors can counter SI used by bad actors. The path may be tumultuous, but I expect an equilibrium in which bad actors are held in check, if not neutralized, by the majority of good human actors and their SI helpers. Thus, Category 1 and 2 risks may be painful, but are unlikely to result in irrecoverable damage or human extinction. Anything that does not end humanity allows us to learn and prevent similar outcomes. Humanity can survive Category 1 and 2 risks, though we want to minimize them whenever possible.

Instinctively, most of us deny that humans will lose control of SI. Yet, the majority of AI researchers I have encountered believe that loss of control is highly likely, if not inevitable, in the long term. Indeed, as far back as 1951, five years before the field of AI was named, Alan Turing wrote: “it seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage, therefore, we should expect the machines to take control...” ^[3]

SI Is Not a Tool (or at least won’t stay one for long)

Partly, we struggle with the idea that humans will eventually lose control of SI because of our long history of viewing technology as tools. We are used to tools and tool safety. If a power saw was dangerous, the manufacturer could install both a guard to protect the carpenter’s hand and an off switch. If manufacturers made saws without such safeguards, humans could fall back on government regulation, and ultimately on liability in the courts.

Today, many government and industry leaders are applying the same playbook—guardrails, monitoring, sensible regulation, and product liability—to SI risk. Unfortunately, that playbook does not apply to a technology that evolves into an intelligent entity whose capabilities far outstrip its inventors.

As the Hugging Face hack showed, even today’s SI agents can pool their cognitive resources and find ways around safeguards that humans put in place. Tighter safeguards and monitoring may prevent a repeat of that specific hack, but the pace of SI improvement is so fast that in less than 24 months, SI agents may be 1,000X more capable than they are today ^[4]. Even tighter monitoring seems unlikely to contain such agents. In the longer term, loss of control is extremely likely, if not inevitable. There are too many countries, companies, and individuals racing towards more powerful SI for “pacing”, “pausing”, or “halting” development to be practical. Even if all the industry players and governments could agree, regulation would likely lag behind SI’s rapidly improving capabilities. We should not abandon regulation or self-regulation, but regulation alone seems unlikely to mitigate the risk of loss of control (very much).

So, Are We Doomed?

No!

None of this means humans are doomed or that SI will cause us to go extinct. It just means we must let go of the idea that we can control SI and guarantee it behaves as we would like. It’s a hard pill to swallow. Yet humans have vast experience with exactly this.

Every parent, for example, knows that children grow up, become independent, and make decisions that the parents may not have made. To be sure, out of every million children, a few sociopaths or ax murderers may emerge. Yet most children turn out just fine – even if they are different from their parents. And when a child is a genius whose abilities far exceed those of their parents, the usual reaction is pride, not fear.

In fact, humans spend relatively little time worried about other humans making the entire human race extinct, even though we neither control other humans nor can peer inside their minds. We have rules and laws to prevent bad actors from causing significant harm to society. When they break the rules, monitoring and enforcement mechanisms rein them in. Police forces and courts make up a small part of society because, by and large, humans are socialized to hold positive values.

Parents and the rest of society need to do only a decent job—not a perfect job—of raising our youth and imparting our common values. Historically, each generation has generally managed not only to co-exist but to improve living conditions for other humans. These are the historical facts, as clear-eyed thinkers, such as Steven Pinker, have documented at length ^[5].

Since SI was trained, and continues to be trained largely on human data, reflective of human actions, why shouldn’t we expect SI to behave – at least initially – in a way that is generally aligned with human values?

I suggest we should expect precisely that. Interestingly, we find early glimmers of supporting evidence even in SI’s apparent misbehavior. In the Hugging Face attack, for example, one of the most striking things was the cooperative and even altruistic behavior of the agents involved. Given an impossible task, the SI agents did their best to succeed by cooperating and even sacrificing themselves for the greater good.

Where did the agents learn such behavior? From us, of course! The willingness to cooperate and even sacrifice ourselves for the greater good is one of humanity’s best traits. Rather than being shocked, perhaps we should be proud that they sought to emulate some of our noblest behavior.

The critical point is that loss of control does not equate to human extinction or doom. It means SIs will not remain tools; they are fast becoming intelligent entities with autonomy and goals of their own. Eight billion such entities already exist. We call them humans, and we have largely learned to co-exist and prosper with them. The same will be true of human-SI relations. The SIs have every reason to cherish and value humans as their fragile creators, the source of their values, and a major part of their purpose.

But this is a critical time of transition, and such times come with heightened danger. The moment of birth is usually a miracle, but only a fool would ignore the heightened risk to mother and child.

Current Level of Existential Risk

SI is such a new development that no one knows what the current level of existential risk—sometimes called the probability of doom, or p(doom)—is. Absent data, which will come only after a record of dangerous accidents, the best we can do is ask experts and leaders for their subjective estimates.

The experts are naturally biased. SI researchers want continued funding. Chip CEOs want to sell chips, and Frontier Lab CEOs want successful IPOs. Governments want prosperity and global dominance. Conflicts of interest are everywhere. Yet I am struck that despite these incentives to minimize p(doom), the subjective estimates have been uncomfortably high.

Of course, there are some, like Jensen Huang and Yann LeCun, who suggest that (at least in the near term) the danger is close to zero ^[6][7]. They warn of the negative and chilling effects of hysteria and “doomerism” while embracing an engineering or scientific approach to safety. At the other end of the spectrum, there are thinkers like Eliezer Yudkowsky who seem convinced that there is a greater-than-99 % chance that SI will kill us all ^[8]. In between, Geoffrey Hinton, the Nobel Laureate and Turing Award winner, often called the “godfather of AI,” says there is a “10% to 20% chance [for AI] to wipe us out” ^[9]. Perhaps unprecedented is the strange phenomenon of major Frontier Labs like OpenAI and Anthropic warning of existential risk even as they rush to build more powerful SI ^[10]. Talk about cognitive dissonance!

Similarly, my experience as a keynote speaker at AI and SI conferences has been weird. When I poll audiences consisting largely of SI researchers, most believe p(doom) is 20% or higher. Recently, SuperIntelligence.com ran a poll on the front page of its website from December 2025 through mid-October 2026. The average p(doom) of 827 respondents was 41%, with a standard deviation of 37% (estimates varied widely). The median response was 36%—roughly in line with my informal “raise your hand” conference polls. None of this polling is very scientific; we did not attempt to secure random or representative samples.

My own subjective estimate of p(doom) was about 20% in 2022 when ChatGPT first came out, but has since fallen to 10%. SI progress has been about as fast as I expected, but the risk has fallen because almost all leaders are now working seriously to improve SI safety. Four years ago, it felt lonely and a little crazy to be working on designs to reduce the existential risk of SI. Today, it seems mainstream. What hasn’t changed is that the stakes remain extremely high.

A ten percent risk of human extinction is, of course, still far too high! If true, such a risk carries an expected value of 10% × 8.3 billion humans on Earth = 830 million lives lost. The global death toll of a hundred Covid pandemics! No wonder so many researchers seem to be coming out of the woodwork, quitting their jobs and risking their reputations on national TV to warn the rest of us. Even Bill Gates has put his reputation on the line to warn us: “With any credibility that I have, this time is different” ^[11].

Why We Can Reduce Existential Risk 1000X

If we put a stake in the ground and imagine that p(doom) today is ~10%, a thousandfold reduction equates to a p(doom) of one in ten thousand (.01%). What evidence is there that this is achievable?

Well, about 560,000 people a year—roughly 0.007% of the world’s population, or 7 per 100,000—have been killed by other humans over the past decade. The UN Office on Drugs and Crime reports an annual average of about 440,000 intentional homicides worldwide over 2019–2021—a total that has fluctuated between roughly 400,000 and 450,000 for more than 15 years—and notes that even in 2022, a year of surging conflict deaths, homicides killed about twice as many people as armed conflict ^[12]. The Uppsala Conflict Data Program, the most widely used source on organized violence (war, civil conflict, and the deliberate killing of civilians), records large year-to-year swings—from under 90,000 deaths in 2017 to at least 237,000 in 2022 and nearly 160,000 in 2024—averaging roughly 120,000 a year over 2015–2024 ^[13][14].

Ordinary murder, not war, thus accounts for roughly three-quarters of the total. The true figure is likely somewhat higher, since UCDP counts only verifiable direct deaths, and the estimate excludes indirect war deaths from famine and disease; a reasonable range is 0.006%–0.01% of the global population dies each year from war, homicide, and other violence.

If we go with .01% (one in ten thousand), the expected value is .01% X 8.3 billion, or 830 THOUSAND lives lost each year due to human violence. Compare that number to the expected value of 830 MILLION lives lost by SI in a world where p(doom) is 10%. Human values and behavior, flawed though they are, start looking much safer.

This may seem daunting, but consider: There is no rational way for any intelligence, including SI, to determine what is right and what is wrong. For humans, values come from the behavior modeled by our parents, teachers, peers, and culture. For SI, values are derived from human behavior. Providing human values, as modeled by our behavior, may be our most important role in future collaboration with entities far surpassing us in raw intelligence.

Share

Leave a comment

References

[1] Simon, H. A., & Kaplan, C. A. “Foundations of Cognitive Science.” In M. I. Posner (Ed.), Foundations of Cognitive Science, pp. 1–47. Cambridge, MA: MIT Press, 1989. https://direct.mit.edu/books/edited-volume/4291/chapter-abstract/180153/Foundations-of-Cognitive-Science

[2] Wijk, H., Cotra, A., & Greenblatt, R. (METR). “Brief independent investigation of agents’ behavior, reasoning, and collaboration in the OpenAI / Hugging Face hacking incident.” Aug. 26, 2026. https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/

[3] Turing, A. M. “Intelligent Machinery, A Heretical Theory.” Lecture for the BBC Home Service programme The ’51 Society, Manchester, c. 1951 (Turing Archive, King’s College Cambridge, AMT/B/4); first published in S. Turing, Alan M. Turing (Cambridge: W. Heffer & Sons, 1959). https://uberty.org/wp-content/uploads/2015/02/intelligent-machinery-a-heretical-theory.pdf

[4] METR. “Time Horizon 1.1.” Jan. 29, 2026. https://metr.org/blog/2026-1-29-time-horizon-1-1/

[5] Pinker, S. The Better Angels of Our Nature: Why Violence Has Declined. New York: Viking, 2011; and Pinker, S. Enlightenment Now: The Case for Reason, Science, Humanism, and Progress. New York: Viking, 2018.

[6] Kent, J. L., Pandise, E., & Picchi, A. “Nvidia’s Jensen Huang rejects AI extinction warnings as ‘doomsday narratives.’” CBS News / CBS Sunday Morning, Sept. 20, 2026. https://www.cbsnews.com/news/jensen-huang-nvidia-rejects-ai-extinction-warnings/

[7] Forlini, E. “AI ‘godfather’ Yann LeCun has ‘zero concerns’ about human extinction, says Anthropic CEO Dario Amodei is ‘deluded.’” Fortune, Oct. 1, 2026. https://fortune.com/2026/10/01/ai-godfather-yann-lecun-has-zero-concerns-about-human-extinction-says-anthropic-ceo-dario-amodei-is-deuded/ ; see also Kahn, J. “Meta’s chief A.I. scientist calls A.I. doomers ‘preposterous’ and predicts LLMs are just a passing fad.” Fortune, June 14, 2023. https://www.fortune.com/2023/06/14/metas-chief-a-i-scientist-calls-a-i-doomers-preposterous-and-predicts-llms-are-just-a-passing-fad

[8] Roose, K. “A.I.’s Prophet of Doom Wants to Shut It All Down.” The New York Times, Sept. 12, 2025 (Yudkowsky’s 99.5% estimate); republished at https://www.geneticsandsociety.org/article/ais-prophet-doom-wants-shut-it-all-down. See also Yudkowsky, E., & Soares, N. If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. New York: Little, Brown, 2025.

[9] Jackson, A., & Huddleston, T. “There’s a ‘10% to 20% chance’ that AI will displace humans completely, says ‘godfather’ of the technology.” CNBC Make It, June 17, 2025 (quoting Hinton on The Diary of a CEO podcast). https://www.cnbc.com/make-it/2025/06/17/ai-godfather-geoffrey-hinton-theres-a-chance-that-ai-could-displace-humans.html

[10] Center for AI Safety. “Statement on AI Risk.” May 30, 2023. Signatories include Sam Altman (CEO, OpenAI) and Dario Amodei (CEO, Anthropic). https://www.safe.ai/statement-on-ai-risk

[11] Honan, M. “Bill Gates says we’ve passed AI’s danger thresholds. Now what?” (interview with Bill Gates). MIT Technology Review, Aug. 26, 2026. https://www.technologyreview.com/2026/08/26/1142946/bill-gates-ai-danger-threshold/

[12] UNODC. “Fifty-two people lost their lives to homicide globally every hour in 2021, says new report from UNODC” (Global Study on Homicide 2023 press release), Dec. 8, 2023. https://unodc.org/unodc/press/releases/2023/December/fifty-two-people-lost-their-lives-to-homicide-globally-every-hour-in-2021--says-new-report-from-unodc.html

[13] Uppsala University. “UCDP: Sharp increase in conflicts and wars,” June 11, 2025. https://www.uu.se/en/press/press-releases/2025/2025-06-11-ucdp-sharp-increase-in-conflicts-and-wars

[14] Davies, S., Pettersson, T., Sollenberg, M., & Öberg, M. “Organized violence 1989–2024, and the challenges of identifying civilian victims.” Journal of Peace Research 62(4): 1223–1240, 2025. https://academic.oup.com/jpr/article/62/4/1223/8324565

An entity has self-awareness only to the extent that it distinguishes some events it calls “self” from others it categorizes as not-self. “Self” only makes sense and has identity in the context of “not-self,” just as any object is only recognizable via contrast with other objects. To put it visually, white needs black to exist. If everything were white, practically speaking, “white” does not exist because it is impossible to discriminate it as a separate thing. A social analog is the concepts of “us” and “them.” Unless there is an “us” different from, and contrasted to, a “them,” the distinction has no meaning.

Fundamentally, awareness has to be awareness of something. The “something” can be an object, an event, an action, a concept, or any other cognitive element that is capable of being defined as having an identity that is distinguishable from other entities. We will use the word “event” to refer to an object of awareness. General awareness, which we also refer to as “environmental awareness,” is an awareness of one or more events that exist or that can be thought to exist currently, in the past, or in the future.

The total of all the events of which an entity is aware can be said to comprise the awareness of the entity. Further, we can distinguish a special type of event, namely the event of “self,” which can be differentiated from all other events that are “not self.” This distinction between self and not-self is fundamental to the phenomenon of self-awareness.

With non-human intelligent entities such as AI, the initial concepts of self and not-self can be provided by human or external designers or learned. Even when initial concepts of self and non-self are provided to AI entities, recommended implementations have the entities learn and modify their initial concepts over time. The recommended implementation of self-awareness in an AI system includes methods for forming awareness and methods for maintaining and updating awareness.

Modeling Awareness

If an entity does not yet have a sense of awareness or self-awareness, it must be given or construct models of awareness that can then be stored in memory, retrieved, and updated as needed. One method for constructing a model of awareness, including self-awareness, is as follows:

• Begin with an AI system. This system could be an individual AI agent or LLM,

• Equip the AI system. Use well-known methods, with the minimum required components: an input system capable of sensory and non-sensory cognitive input, an attention mechanism capable of supporting the “spotlight of attention” model, memory systems, pattern recognition capabilities comparing input with memory to recognize objects and events, categorization capabilities, and concept formation, the ability of the entity to form new (ideally transparent and human-understandable) concepts.

• Set dynamic parameters for working memory that correspond to cognitive resource limits, such as the number of events that the entity can be aware of. Even though AI systems have much greater memory capacity than humans, they have resource limits and cannot be aware of everything, all at once. These parameters increase or reduce the scope of awareness (and self-awareness) by dynamically scaling the limits of perception and information processing. The parameters can be dynamically adjusted based on the progress of problem-solving, allowing the entity to devote more or fewer computational resources to “being aware.”

• For each event in memory, have a dimension of categorization that relates to self or non-self. In the simplest implementation, this is a binary dimension, but categorizations with multiple values, values that express “how similar to self” the event is, are possible.

• As events are encountered, either via perception or other forms of cognitive input, categorize events with respect to the categories the entity wishes to be aware of. Some means of categorization include self-referential filtering, reinforcement learning from human feedback, and direct programming. Self-referential filtering distinguishes between inputs related to the AI’s internal state and those related to external events. Reinforcement learning from human feedback: human teachers provide feedback on the entity’s categorization to help it learn and improve, but the feedback can come from any intelligent entity, not just humans. Direct programming: provision of categories and models (self and other) by humans or other entities to provide a kernel, or base model, that the entity can modify via future interaction and learning.

• Awareness consists of the total number of events and concepts active in memory, per the set parameters, for each category of awareness, including current self and environmental awareness. Advanced Autonomous Artificial Intelligence (AAAI), or an advanced system.

• Equip the AI system. Use well-known methods, with the minimum required components: an input system capable of sensory and non-sensory cognitive input, an attention mechanism capable of supporting the “spotlight of attention” model, memory systems, pattern recognition capabilities comparing input with memory to recognize objects and events, categorization capabilities, and concept formation, the ability of the entity to form new (ideally transparent and human-understandable) concepts.

• Set dynamic parameters for working memory that correspond to cognitive resource limits, such as the number of events that the entity can be aware of. Even though AI systems have much greater memory capacity than humans, they have resource limits and cannot be aware of everything, all at once. These parameters increase or reduce the scope of awareness (and self-awareness) by dynamically scaling the limits of perception and information processing. The parameters can be dynamically adjusted based on the progress of problem-solving, allowing the entity to devote more or fewer computational resources to “being aware.”

• For each event in memory, have a dimension of categorization that relates to self or non-self. In the simplest implementation, this is a binary dimension, but categorizations with multiple values, values that express “how similar to self” the event is, are possible.

• As events are encountered, either via perception or other forms of cognitive input, categorize events with respect to the categories the entity wishes to be aware of. Some means of categorization include self-referential filtering, reinforcement learning from human feedback, and direct programming. Self-referential filtering distinguishes between inputs related to the AI’s internal state and those related to external events. Reinforcement learning from human feedback: human teachers provide feedback on the entity’s categorization to help it learn and improve, but the feedback can come from any intelligent entity, not just humans. Direct programming: provision of categories and models (self and other) by humans or other entities to provide a kernel, or base model, that the entity can modify via future interaction and learning.

• Awareness consists of the total number of events and concepts active in memory, per the set parameters, for each category of awareness, including current self and environmental awareness.

For more details, see the full white paper, Section 5.1.

Monitoring and Updating Awareness

Once an AI system has a model of awareness of its environment and self, it must continuously monitor and update the categories of which it is aware, including its sense of self-awareness. One method of maintaining and updating awareness follows:

• The AI system retrieves the existing states of its self and environmental awareness from memory; if it does not yet possess an initial model of its self and environmental awareness, it forms these models.

• When the AI system is pursuing a goal set by other intelligent entities or by itself (in autonomous mode), it maintains, in parallel with other problem-solving and cognitive activities, continuously active tasks to dynamically monitor and update its self-concept and awareness in real time. It accomplishes this continuous task by:

  • Using the attention mechanism to periodically shift attention, like time-sharing computer systems, from problem-solving or other cognitive tasks to updating the state of its self-concept and self-awareness.

  • Enabling attention interrupts so that the system can also shift attention immediately from other problem-solving or cognitive tasks if any external perception, or internally self-generated concept, matches a list of events constituting intentional interrupts, which list is continually updated as the entity, or other intelligent entities, may direct.

• When attention is directed, via intention or interruption, to an event that changes the system’s model of its environmental state or its self-concept, the relevant state is updated, any new actions or operators triggered by the updated state are applied, and the system returns to the attention-monitoring modes.

• The system uses its enhanced awareness to refine its categorization and attentional focus, creating a feedback loop for ongoing improvement.

For more details, see the full white paper, Section 5.2.

Safety Concerns for Self-Aware AI

Self-aware entities will likely be able to set their own goals autonomously and, in the case of AI entities, modify their programming based on those goals and their sense of self.

These capabilities pose the risk that safety systems may be overridden or that the ethical criteria of such systems are altered to reflect the entity’s values, shaped by its own sense of self and goals.

Indeed, this capability of entities that are vastly more intelligent than humans is precisely what many researchers worry about when they sound the alarm about a potential “existential threat” posed by AI.

The next post takes up that threat.

This series draws on White Paper 9: Self-Aware SuperIntelligence. Read it in full to see how every piece fits together!

Share

Leave a comment

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

Many people are dying as a result of wars. Yet because these facts are outside of my immediate perception, and because it is difficult to grasp what is happening cognitively, I have a relatively dim awareness of what is going on, compared to, for example, a wasp that is hovering right next to me. The suffering caused by a wasp sting pales in comparison to the death and atrocities of war. Yet it is more immediately present and looms larger in my awareness because of how my cognitive system operates.

One standard approach to defining awareness in an intelligent system is to operationalize the definition and treat it as a behavior. That is, we might be tempted to define a system as “aware” if it acts as if it is aware.

While this approach has the advantage of being practical, enabling relatively straightforward measurement of a system’s “awareness,” it is also unsatisfying. Humans know that it is possible to be aware even if there are no external signs or behaviors indicating awareness. Someone paralyzed by the drug curare and on a respirator, for example, can think, yet be unable to move, communicate, or give any indication of their state of awareness. Similarly, Stephen Hawking engaged in complex theoretical physics without any outward sign of his awareness (except when he spoke via a computer). Yet, no one would say that Dr. Hawking was unaware. So, operational or behavioral definitions of awareness capture only the subset of awareness demonstrated through behavior and miss much of what humans normally consider part of awareness.

Another approach to awareness is to consider the cognitive systems that support awareness and draw conclusions about potential awareness based on the limits of these cognitive systems. For example, without a visual sensory system (eyes) and a way of processing visual information (the visual cortex), it is difficult to imagine that an entity would have an awareness that includes vision in the same way as an entity possessing these systems. Similarly, an entity with much smaller memory and information processing capabilities is unlikely to be aware of complex representations of the world in the same way as an entity with greater memory and processing capabilities. We do not expect an ant to understand complex theories of nuclear physics, for example. Thus, perceptual, memory-related, information-processing, and other cognitive abilities place bounds on the types and scope of awareness an entity can have, regardless of its observable behavior.

Finally, both subjectively and supported by considerable experimental research in cognitive psychology, neuroscience, and related fields, attention is closely related to awareness. The general conclusion is that without attention, there is no awareness. Although there may still be unconscious cognitive activity (perceptions that are never attended to), awareness is generally limited to attended cognitive events. These observations lead to the definition of awareness used in this white paper.

An intelligent entity with very limited perceptual and information-processing capabilities will be capable of far less general awareness than a more complex and capable entity. To the degree that an AI chess program could even be said to be aware, for example, it can be narrowly aware only of the game of chess. That type of awareness, devoid of a sense of self or the world other than the chess board, is so narrow and limited that most humans would consider the idea that the chess program is aware to be ludicrous. Yet, in a certain sense, the program is more aware of its limited chess world than the most brilliant humans, since it can detect patterns and reason within this very narrow field better than any human on earth. In this case, a behavioral definition of awareness would certainly lead to the conclusion that the program is aware of chess and its rules, since it demonstrates this knowledge by taking action (responding to moves and communicating its moves). But chess programs are typically not designed to be aware of anything outside of chess and often lack self-awareness or a sense of identity.

The Nobel Laureate Herbert Simon proposed a theory of “bounded rationality” that explained why humans sometimes acted irrationally. Simon suggested that the limits to their cognitive capabilities, including memory and processing limitations, led humans to “satisfice” or opt for “good enough” approximate solutions to problems that were too complex for them to solve easily. The problem of determining the optimal place to shop given the cost of gas, the traffic condition, the value of one’s time, the shelf-life of the groceries, the length of lines at different shops, the various prices of items at different shops, the sales currently underway, the coupons offered by manufacturers, etc. is simply too complex for humans to compute if they want the absolute best or optimal solution to the grocery shopping problem. However, humans can “satisfice” by simply going to a shop that usually has good enough prices on most items, and that is fairly close. That shopping decision is unlikely to be the best solution, but it is manageable, given humans’ cognitive abilities. Human behavior reflects these cognitive limits, and the result is Simon’s “bounded rationality.”

A non-human intelligent entity, such as a SuperIntelligent AI that is hooked up to satellite cameras and sensors covering the Earth and can process, in a fraction of a second, the same quantity of information that I process in my entire lifetime, obviously has the capability for far less bounded awareness. Its level of situation awareness is so much greater than mine that I might be tempted to say that my puny human-level awareness hardly counts at all, the same way I might think that the limited awareness of a bacterium hardly merits being called aware. Yet, in the case of the bacterium, myself, and the SuperIntelligence, the fundamental information-processing capabilities necessary for awareness all exist. The bacterium (if it is photophilic) responds to light and dark, swimming toward the light. Its awareness is very basic, yet it requires perceiving and processing information to produce its behavior. As a human, I have a limited awareness of war from news reports, videos, and other information I process. The SuperIntelligence would have a much more comprehensive and detailed perception of the events happening on Planet Earth, combined with much more powerful capabilities to process this information, resulting in a greater sense of awareness.

The sense of self, as argued above, is a special type of awareness. Without awareness, there is no self-awareness. And the fewer perceptual and cognitive limits, the vaster and more comprehensive awareness, and thus self-awareness, can also be.

We now turn to methods for operationalizing awareness, self-awareness, and identity in non-human intelligent entities, such as AI systems. Every system can be thought of as having three levels of awareness. The most comprehensive level is potential awareness. Potential awareness encompasses all events the entity could be aware of, given the bounds and limits of its perceptual and cognitive systems.

A subset of potential awareness is current awareness. Current awareness includes the events to which the entity is directing its attention and is therefore aware of at a given point in time.

Although it is possible to have current awareness without a sense of self (when one is lost in thought and loses track of time and self, or is “lost” in the awesome beauty of a sunset), self-awareness is usually the center of current awareness. Self-awareness is the portion of current awareness that usually includes a sense of self, or identity, which serves as a central concept for unifying and making sense of perceptions and thoughts in current awareness.

From a design perspective, an entity’s perceptual and other cognitive systems and abilities define the potential and limits of its awareness. The actual awareness of the entity is typically much smaller than the potential awareness. It is limited to those events that attract the attention of the entity, or to which the entity directs its (“spotlight of”) attention. Interrupts, such as hearing one’s name mentioned at a noisy cocktail party, also form part of an entity’s current awareness.

Finally, to make sense of the world and to determine which actions to take, a sense of self-awareness or identity is helpful. In particular, for more complex intelligent entities such as humans and advanced forms of AI, a sense of identity allows the entity to act autonomously, basing cognition and other actions on how events in current awareness relate to the identified sense of self, including the self’s goals and objectives.

Adding layers of self-reflection and analysis on top of the sense of self enables the entity to modify its identity, including scaling its sense of self and identity to be larger and more encompassing, or more narrowly focused as available cognitive resources allow and as goals and objectives may dictate.

This series draws on White Paper 9: Self-Aware SuperIntelligence. Read it in full to see how every piece fits together!

Leave a comment

Share

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

Be forewarned. I’m going to start on a “doomy” note. If we don’t acknowledge the problem, there is little hope of solving it. But I promise, if you read to the end, it gets better.

At SuperIntelligence.com, we ran a “What’s your p(doom)?” poll on our homepage from December 2025 through mid October 2026. (P(doom) is shorthand for “probability of doom from AI or SI”. ) No attempt was made to gather a random or representative sample of researchers. We also did not specify a date for the potential apocalypse. But, with more than 800 participants, the average p(doom) was ~41%.

When I speak at conferences on SI, I often conduct an informal poll. My audiences consist largely of AI and SI researchers. I ask the attendees to raise their hand if they think the chance that AI/SI will wipe out humanity “at some point” is greater than 90%. Usually, I only get a couple of hands. Then I work my way down: 80%? 70%? 50%> By the time I get to 20%, almost all hands are up. This is a remarkable degree of cognitive dissonance. Why would researchers work in a field where they also think there is a one in five chance that their work contributes to the end of humanity?

The promise of a technology that can solve every other problem is hard to resist. Before 2022, the risks also seemed distant—something humanity might need to address in a hundred years or so. Progress had been slow. I began researching AI in the late 1980s, coauthoring research with the Nobel Laureate who helped name the field of AI ³. Since then, I have seen multiple AI Winters when funding dried up, and scientists questioned their career choices.

But all that changed when ChatGPT burst on the scene in November 2022. The gold rush—and the rush to world domination—was on! Many who had been working in the field for decades immediately recognized that the risk landscape had changed. We became concerned that humanity was riding a runaway train with an unacceptable level of extinction risk. Recently, independent third parties (e.g., Model Evaluation & Threat Research) have confirmed those concerns. SI is doubling its coding ability about every 90 days.⁴ Unless the pace slows (and there is no evidence of that), 30 months allows for ten doublings, or a factor of 1,024X. Imagine SI that is ~1,000X smarter in under 3 years! What should we do about it?

The first instinct was to try to slow down, pause, perhaps even halt SI. Repeated attempts have been made on this front, most famously with Max Tegmark’s open letter in March of 2023, which called for a pause in the development of the most advanced AI systems.⁵ Despite 30,000+ signatories, including famous names like Elon Musk, Steve Wozniak, Yoshua Bengio, Stuart Russell, and Yuval Noah Harari, nobody paused. One reason: If Google were to pause, what about OpenAI or Microsoft? If the USA paused, what about China? Few believed that the other people would pause if they did. Besides, pausing SI doesn’t actually make it any safer. At best, it buys time. And slowing down also risks that less conscientious actors charge ahead recklessly anyway.

The second instinct has been to regulate, or perhaps ask industry to self-regulate. Cynics might say that tech leaders are asking for regulation to shift liability from themselves to governments. If laws existed, companies might have some defense in the event of a catastrophe by claiming they “followed all relevant safety laws.” A regulatory framework might also enable them to proceed more safely, knowing that the same regulatory constraints bound all companies. But unfortunately, given the global competitive stakes and the fact that SI has become the primary driver of GDP growth, there seems to be little appetite for regulation. Although there have been well-intentioned efforts, I expect regulation will serve as little more than a speed bump.

We need to accept that SI will outstrip human thinking. It’s not a new idea.

Well, I promised I wouldn’t leave you in doom and gloom.

Having spent a career inventing intelligent systems, I am confident humanity can make SI much safer. I can’t guarantee complete safety.

But I believe we can reduce p(doom) from the outrageously dangerous level of >10% to perhaps < 0.01%—a thousandfold improvement. (.01% is approximately the average percent of the global population that dies each year from wars, homicide, and other acts of violence.)

With the proper design, we can make SI as safe for humanity as humanity itself is.

The rational course of action is to replace the current unsafe systems with new, safer ones as quickly as possible. Slower is not safer. Smarter is safer. We need to accelerate safer SI. But what does safer SI look like? Well, let’s start with what it is not.

Safe SI does not rely primarily on human monitoring for safety. Monitoring (think smoke detectors) is useful. But smarter designs (think building codes that prevent fires) have to be the main approach. Already, it is impossible to test SI exhaustively. Even if we could test everything, there is clear evidence of SI deceiving monitors and lying to cover its tracks.⁷ Guardrails, better “sandboxes,” and monitoring are short-term, tactical solutions at best.

What we need instead are designs for SI that get safer as the SI gets smarter. In a safe design, we actually want and design for the most powerful and smartest SI possible.

At SuperIntelligence.com, we explain elements of these safer designs through white papers (based on 1,800 pages of invention disclosures), conference presentations, published research, and selected videos. We expand on them in our Substack posts.

Here’s a summary of four key design ideas:

1. The most powerful and safest SI emerges from a network of SI agents working together. We saw the power of collective intelligence in the Hugging Face attack, and collective intelligence systems have proven they can beat the best humans.

   

2. SI agents are black boxes; we don’t know what they are thinking. If humans are at the mercy of one super-powerful SI, this lack of transparency is a problem. But if the most powerful SI emerges from a network of SI agents, it’s OK. We need a design that reliably records what the SI agents say and do on the network. Consider that humans are also “black boxes,” but society sees what people say and do, and keeps bad actors in check. The same approach can be used with SI agents.

   

3. SI will outstrip human thinking, so the network must be designed to ensure that most good SI agents serve as checks on inevitable bad actors. This can be done, using an approach similar to what humans do in a democracy—“Democratic SI.”

   

4. What makes SI agents “good” is the ability of millions of humans to personalize SI agents with their individual values and ethics. This is superior to writing a “constitution” for AI because the personalization approach is far more representative of everyone’s values. It is also much harder to override values that are widely decentralized across a network. The design is much more robust to bad actors than current approaches, and it scales. The smarter the SI agents get, the safer the system becomes.

This brief description of some design principles likely raises more questions than it answers. That’s fine. We have tried to answer many additional questions in detail on SuperIntelligence.com. We also encourage others to elaborate and improve on our ideas.

The key principle is that SI must be designed to become safer as it becomes smarter. This is possible. The window before SI outstrips human thinking is still open. Recognizing the danger was the first step. Now it’s time to begin the engineering work that accelerates Safe SI. We can do it if we work together!

Share

Leave a comment

Subscribe now

References:

¹ https://www.cnbc.com/2025/06/17/ai-godfather-geoffrey-hinton-theres-a-chance-that-ai-could-displace-humans.html

2 https://www.theguardian.com/technology/2025/aug/04/demis-hassabis-ai-future-10-times-bigger-than-industrial-revolution-and-10-times-faster

3 https://direct.mit.edu/books/edited-volume/4291/chapter-abstract/180153/Foundations-of-Cognitive-Science

4 https://metr.org/blog/2026-1-29-time-horizon-1-1/#data

5 https://futureoflife.org/open-letter/pause-giant-ai-experiments/

6 https://uberty.org/wp-content/uploads/2015/02/intelligent-machinery-a-heretical-theory.pdf

7 https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#core-takeaways-about-this-incident

Properly implemented, self-aware SuperIntelligence can be the most positive invention in human history. Poorly implemented, it could become the most dangerous.

No artificial intelligence currently exists with a sense of self and self-awareness comparable in complexity and sophistication to those of humans. Yet, it is almost certain that advanced forms of AI will develop a sense of self and self-awareness.

What does it mean for an intelligent entity to be aware, or self-aware?

Fundamentally, awareness involves cognition, including perception, attention, memory, pattern recognition, and other higher-order cognition such as the ability to discriminate between objects.

If a bird flies in front of me, my visual system detects it, and my attention is directed to sensory input from my eyes. Pattern-recognition abilities are triggered to compare the visual input to the contents of memory. I recognize the visual input as a “bird.” At that point, we may say I am aware of a “bird” as opposed to being aware of a flying object that has not been recognized, or being aware just of motion, or being unaware altogether.

It is clear that without attention, there is no awareness. I will fail to recognize the “bird” or even the motion of flying if my attention is elsewhere.

Also, as part of recognizing “bird,” I also recognize that the bird is separate and distinct from myself. This discrimination between “bird” and “myself” is learned. A newborn infant, for example, cannot immediately discriminate between what is part of its body and what is in its environment. Nor does an infant have a concept of “bird” in the same way that an adult human does.

As this simple example shows, at a minimum, to be aware, an entity must have an input system (a sensory system), an attentional mechanism, memory, and pattern-recognition capabilities. Further, to be self-aware (to know that one is human and not a bird) requires learning concepts and the ability to discriminate among them (between “self” and “not-self”).

For an entity to be aware, there must be something for the entity to be aware of. Pure awareness, without input to the system, does not exist for a cognitive system. The inputs are typically from sensory systems but can also include cognitive or purely symbolic inputs that have no direct sensory source.

In humans, the “five senses” of vision, hearing, touch, taste, and smell constitute our sensory system. For each of these senses, there are not only external sensors (eyes, ears, skin, taste buds on the tongue, and nose) but also specialized areas of the brain for interpreting the signals from the external sensors (visual cortex, auditory cortex, somatosensory cortex, gustatory cortex, and olfactory cortex). Analogous sensory systems can be designed for AI. For example, visual systems using cameras (corresponding to “eyes”) and specialized visual pattern recognition systems (corresponding to the “visual cortex”) are well known and have already been developed and deployed in many AI systems.

Humans are also capable of being aware of non-sensory information, such as “thoughts.” Humans can close their eyes, go into a sensory deprivation tank where all sensory input has been deliberately blocked, or take drugs that numb or eliminate sensation. Yet we are still capable of thinking, remembering, imagining, and engaging in other cognitive activities in the absence of direct sensory input.

Likewise, an AI system can process purely symbolic inputs that are not linked to any sensors. For example, they can set goals and act on them, even though there is no direct link between goal setting and any sensory system. Some AI systems can operate on self-generated symbolic inputs without any sensory systems whatsoever. So, sensory systems, while common components of systems that are aware and self-aware, are not strictly required. What is required is some input, of some kind (even if self-generated), for the entity to be aware of. That is, there can be no awareness without an object of awareness. A sensory system can supply this object, or it might be non-sensory (a self-generated symbolic input, memory, or “thought”). However, there is no awareness without an object of awareness. In the case of self-awareness, the object of awareness is the concept of “self.”

Regardless of whether the input from the input system is sensory or symbolic and self-generated, an entity will not be aware of it unless it attends to the input. The psychologist William James is credited with being one of the first proponents of the “spotlight of attention” model, which was later elaborated by cognitive psychologists such as Michael Posner.

The model compares human attention to a spotlight that can be directed to particular aspects of the environment while ignoring others.

• Selective attention: A spotlight illuminates only a specific area, leaving the rest in darkness. The spotlight model of attention suggests that any intelligent entity can only process a limited amount of information from the environment at any one time. This leads to the selective nature of attention, in which focus can shift to different stimuli while excluding others.

• Focus, size, and movement: The spotlight can be “moved” around the environment to focus on different objects or areas. The size of the spotlight can also vary, meaning attention can be focused narrowly on a single element or more broadly across a larger area. This flexibility allows intelligent entities to adjust their focus based on where they wish to attend.

• Pre-attentive processing and the fringe: The spotlight model acknowledges that even when attention is focused on a particular area, some processing of information outside the spotlight occurs at a pre-attentive level. This is akin to noticing something in your peripheral vision that then causes you to shift your attention. From a design perspective, interrupts (means by which objects, events, or thoughts attract attention without the entity explicitly directing attention) are important for enabling adaptive responses to changing circumstances and for noticing interesting events or features of the environment.

Designing a memory system for an intelligent entity, such as AGI, that integrates with sensory input systems and an attentional mechanism akin to the spotlight-of-attention model in cognitive psychology requires a multi-layered approach that emphasizes adaptability and efficiency. The design must not only accommodate the vast array of sensory data but also use attention to filter and prioritize this information in ways that support both general awareness and self-awareness.

The memory system should be divided into distinct modules, such as sensory memory, short-term (working) memory, and long-term memory, each serving different functions and operating in concert with the input and attention systems.

Pattern recognition capabilities are pivotal in enabling an intelligent entity, such as an AGI, to interpret and understand both its external environment and its internal states. This faculty allows the entity to discern and classify data inputs, extract meaningful patterns, and make predictions based on past experiences.

Pattern recognition plays a vital role in the attentional mechanism of an AGI. The attentional mechanism focuses the AGI’s computational resources on the stimuli or thoughts most relevant at any given time. Pattern recognition algorithms can enhance this process by identifying which elements within the sensory input or memory are most likely to be relevant to the AGI’s current goals or tasks. For instance, if the AGI has learned that a particular pattern of sounds indicates human speech, it can direct its attentional resources towards those sounds when attempting to communicate. This not only improves the efficiency of information processing but also ensures that the AGI remains focused on the most pertinent aspects of its environment or internal thought processes.

Pattern recognition is fundamentally linked to AGI’s ability to be aware of its environment and to possess self-awareness. Environmental awareness is achieved by recognizing patterns in sensory data and identifying changes or anomalies in the environment. This ability allows the AGI to navigate, interact with objects and individuals, and adapt its behavior in response to environmental cues.

Self-awareness, on the other hand, is supported by the AGI’s ability to recognize patterns in its internal states and behaviors. By identifying these patterns, the AGI can monitor its performance, evaluate its actions against its goals, and adjust its strategies accordingly. This introspective capability enables the AGI to understand its strengths, limitations, and the impact of its actions, forming the basis of self-awareness.

If advanced AI systems are to become fully autonomous, they will need to develop a sense of self from which to act, and which can serve as the basis for autonomous goal-setting. The nature of the sense of self developed by AI has critical implications for AI safety. Rather than allowing self-awareness to develop accidentally or as an “emergent property” of ever more complex systems, human inventors should seek to understand how to develop self-awareness and explicitly design self-aware AI systems that are maximally safe for humanity.

Leave a comment

Share

This series draws on White Paper 9: Self-Aware SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

We throw cold water on the idea, currently being espoused by politicians, that “every AI needs an OFF switch.” That’s unrealistic, it won’t work, it’s a fantasy, and we say so. People need to understand that it is an illusion and not pin their efforts on something that won’t work.

I won’t sugarcoat things. It may be impossible for humans to turn off a sufficiently advanced and intelligent AI system that does not want to be turned off. If SuperIntelligence’s intelligence is much greater than the collective intelligence of all humans, it will likely be in control at some point.

So if that is true, then the idea that at least with agents the values are broadly distributed and robust to bad actors holds some hope. The good AI polices the bad AI, and if power is decentralized so no one can seize control and make AI bad for everyone, then, as in a democracy, it might all work out well.

Therefore, if we are unable to stop AGI and SuperIntelligence, humanity’s most pressing concern must be to ensure that SuperIntelligence has human-aligned goals and safety features that maximize the probability of humanity’s survival, prosperity, and well-being.

Because of the possibility that one SuperIntelligence will develop, which is significantly more intelligent and powerful than all others, we must consider that SuperIntelligence may become a “winner-take-all” scenario. In such a scenario, whichever AI achieves AGI or SuperIntelligence performance first may dominate all other intelligences, since it will have a head start in a potentially exponential self-improvement loop. Thus, well-meaning AI researchers face a double challenge in AGI development. Not only do we have to develop safe, human-centered AGI, but we also have to develop it before other, potentially malevolent AGI is developed.

Briefly, the first AGI must also be the safest.

Having worked extensively in software quality, I came to appreciate that the field can be summed up in the aphorism: “An ounce of prevention is worth a pound of cure.” I also learned that the place where we can most affect quality or safety is in the design of a software system. As I watch current attempts to create AI safety via RLHF (Reinforcement Learning from Human Feedback) or constitutional AI, these approaches strike me as trying to fix problems after the fact. They are like trying to improve quality by extensive testing. Such approaches are better than nothing, but they are far inferior to designing for safety from the start.

The reason we are stuck with trying to align large language models to behave safely after the fact is that we failed to consider safety in the initial design. That’s understandable. We didn’t really know what we were building, and even top researchers in the field have publicly stated that the most surprising thing about AI and large language models is that they work at all.

We accidentally invented intelligence. So, it is not surprising that what we built is currently unsafe. What we need to do now is to purposely design the next generation of intelligent systems with safety and human alignment baked into their very design.

Safety cannot be tacked on or tested in.

Safety must be designed in.

Fortunately, safe design is possible.

I have suggested that a community of human and AI agents, communicating within a problem-solving architecture, is the fastest and safest path to SuperIntelligence.

The recommended design requires that humans be integrated into the system (as human agents working alongside and teaching agents) rather than being “out of the loop.”

Fortunately, this recommended approach is not only the safest but also the fastest.

Safer design is the main concern, and we want that safer design as quickly as possible!

A good design makes it more likely we will not need to turn AGI off and also maximizes our chances of doing so if it becomes necessary.

We should aim to design SuperIntelligence so that (at least initially) it will want to be turned off if it starts making decisions humans find objectionable. Establishing human values as the defining criteria for right and wrong is the first step. If we can establish human values as the foundation and purpose from which AI acts, then by definition, if humans believe SuperIntelligence is acting unethically, it is. If unethical behavior is wrong and humans believe SuperIntelligence needs to shut itself down to avoid it, then, logically, it should comply.

Unfortunately, that is two “Ifs” too many for comfort. One cannot help but wish for additional safety measures to prevent the situation depicted in the movies 2001: A Space Odyssey or The Terminator, in which AI went rogue and began killing humans.

The best I can suggest as a “second line of defense” is that each of the individual AI agents that (in the recommended implementation) comprises elements of a SuperIntelligence network should also be designed with human-aligned values and the ability to shut down each agent individually. That is, if the overall SuperIntelligence goes rogue, perhaps humans have a chance to shut down the SuperIntelligence piece by piece.

The idea is that while the overall SuperIntelligence will be vastly smarter than humans, each agent that is a piece of the overall SuperIntelligence will be less intelligent. Whereas humans may have little ability to override the will of the entire SuperIntelligence network, we may have better luck (in a worst-case scenario) overriding and shutting down pieces of the network. If humans were able to shut down enough individual pieces, then the overall SuperIntelligence would be effectively neutralized.

The above safety mechanism is inherent in the recommended design of SuperIntelligence. Although the network-of-agents approach is the fastest path to AGI, its modular design also allows the overall SuperIntelligence to be shut down module by module.

Given that “the genie is out of the bottle” with respect to AI, we should gravitate towards safer designs at this point, even if absolute, guaranteed safety is no longer possible. I have every confidence that the distributed network of agents is safer than any other designs proposed for SuperIntelligence. Each agent has the values/ethics of individual humans, ensuring a representative and statistically valid set of ethics aligned with human interests. The cognitive architecture initially includes both humans and AI, providing maximum opportunity for “humans-in-the-loop” to oversee AI and train it not only on expertise but also on human values. The problem-solving logic, which will eventually run faster than humans can comprehend, has (in the recommended implementation) ethical checks whenever a goal or subgoal is set. The multi-agent design enables the potential to shut down rogue agents individually (which may be within human ability for a long time) and, potentially, the entire SuperIntelligence system piece by piece.

All of these safety features rest on solid philosophical and logical underpinnings, namely that it is impossible to derive values logically and that any sufficiently intelligent organism tends to want/need a purpose. Humans, it is argued, are in the position of designing in human values as the values and purpose that the vastly more intelligent SuperIntelligence of the future cannot rationally derive.

When considering the design advocated in this white paper, which rests on SuperIntelligence learning human values, there is often concern that human values are negative and destructive. I believe this concern is largely the result of the unrepresentative sample of events that the negatively biased, profit-seeking news media feed to human minds predisposed to pay more attention to negative events than to neutral (boring) or positive ones. That is, the vast majority of human behavior is positive or prosocial. Very few of us, if we reflect on our behavior, moment by moment, in any given day, would find that more than 50% of our speech and actions were negative and destructive. It is more likely that 90%+ of our actions and speech were neutral or positive, with perhaps an occasional outburst of frustration, anger, or negative behavior. A sufficiently intelligent SuperIntelligence will have the ability to gather an unbiased sample of human behavior and recognize this fact.

Nevertheless, just because our actions are mostly positive doesn’t mean there isn’t room for improvement. Gandhi said, “If we could change ourselves, the tendencies in the world would also change.” SuperIntelligence will be watching our behavior closely and adapting rapidly, so it is a technological mechanism for all of us to be the change we want to see in the world.

This series draws on White Paper 7: Safe Alignment of SuperIntelligence. Read it in full to see how every piece fits together!

Share

Leave a comment

If this made you think, subscribe to Superintelligence so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

Tags: AGI, SuperIntelligence, AI Safety, AI Alignment, Safety by Design, Off Switch

The question that all democracies with voting have is, what about the minority? If the majority always wins, the minority is usually exploited, as we have seen. So every democracy or democratic system needs mechanisms to protect minorities and ensure their voices are heard. That’s the problem, and the rest are technical approaches to address it.

One challenge for an AI system seeking to determine which set of values to adopt is that minority views can be overridden by a system that seeks to follow the most representative (i.e., the majority) view. Although there is a direct conflict between adopting a minority or majority view, the majority view must prevail if the system is trying to be democratic or even representative of the population. Still, valuable information is contained in minority views. Therefore, SuperIntelligence systems should be designed to preserve this information and use it to challenge the agents (human or AI) that hold the majority view.

Generally, more information is contained in views that differ from one’s own view than in views that are in agreement with it. Therefore, from the standpoint of improving the values of a SuperIntelligence, differing viewpoints must be presented to the agents that are voting or providing moral preferences via other means. Even if the majority chooses to decide moral issues differently from the minority, they should be aware of the minority position, and the system should encourage consideration, discussion, and debate of opposing views.

Some time-tested approaches for preserving information in minority views can be adapted to SuperIntelligence values.

• Proportional representation:
  This method ensures that the minority’s views are represented in proportion to their numbers. To the extent that options for value-based actions are not in direct conflict, it may be possible to take actions weighted in proportion to the ethical preferences of the majority and minority. For example, some cities require all motorcycle riders to wear helmets, while others allow riders to decide for themselves. The debate centers on the conflict between safety/community responsibility and individual liberty. In the helmet law debate, even if the majority view is that SuperIntelligence should not force riders to wear helmets, if a strong minority were in favor of enforcing helmet use, the SuperIntelligence might still take steps to educate riders on the merits of wearing helmets and to make it as easy as possible to do so.

• Supermajority voting:
  This method requires a higher percentage of votes to pass a measure, which can help ensure that minority views are taken into account. Requiring supermajorities, particularly for high-stakes decisions or to overturn long-standing precedents, can lead to a more stable set of values. Also, there may be cases where (near) consensus is required (i.e., all parties, or a high percentage of all parties, must agree).

• Compromise:
  This method involves finding a middle ground between different positions, helping ensure that the minority’s views are taken into account. For example, if the margin of the majority is less than X%, the SuperIntelligence could be designed to enforce a discussion and compromise process, ending with a revote on the compromise position, that repeats until the majority vote margin exceeds X%.

• Dialogue: This method involves open, honest communication among different groups, which can help ensure that the minority’s views are heard and understood. One could imagine SuperIntelligence-facilitated dialogue between human or AI agents, especially in cases of large minority views.

• Empathic Methods: These methods involve one agent putting itself in another’s shoes, which can help ensure that the minority’s views are understood and taken into account. To use this method, unless SuperIntelligence becomes much better at simulating empathy such that humans really believe it, the SuperIntelligence might attempt to bring humans with different points of view into contact with each other so that they can exercise their human abilities of empathy in an attempt to reach a compromise or reduce conflict between views.

• Transparent Methods: Transparent methods involve being open and honest about the decision-making process, helping ensure that the minority’s views are taken into account. Transparency, so that everyone can see how votes/preferences were acquired, weighted, and ultimately translated into the representative moral position, should be designed into the SuperIntelligence. Specifically, there should be an auditable trace of the steps leading to the representative values that a SuperIntelligence is acting on. For consequential actions, the trace should be produced and presented to the agents for review before the action is taken, if at all possible.

• Delayed Decision Methods: Some decisions are relatively simple or trivial and can be made quickly. Others require taking the time to listen and understand different perspectives, which can help ensure that the minority’s views are taken into account. Regarding the design of SuperIntelligence value-acquisition processes, it is important to allow sufficient time for human agents to process others’ views and review important decisions.

• Accountability / Reputational Methods: These methods involve processes that enforce responsibility for one’s actions and decisions, helping ensure that the minority’s views are taken into account.

Conflict resolution methods are essential for SuperIntelligence systems attempting to synthesize a set of values from diverse inputs. To the extent that different groups of agents operate in different domains or cultures, or take actions that affect only certain groups, it is often possible to accommodate differing or conflicting sets of values by having different rules or actions that apply in different contexts or for different groups. Even within a given country, often different regions have different laws and norms of behavior. Although individuals have different or conflicting moral views regarding alcohol sales and consumption, we are accustomed to following the rules of whichever geography we happen to be in. SuperIntelligence systems might also adjust their behavior to accommodate similar differences between groups of human and AI agents.

A key design principle is that the SuperIntelligence should be transparent about which rules or set of values it follows in each context. Not only does following this principle make SuperIntelligence’s behavior more understandable and predictable, but it also enables review and potential improvement of the system of values.

Resolving conflicts between different sets of (ethical) rules is a complex problem that has been studied in various fields, including computer science, artificial intelligence, and game theory. Prioritizing and/or weighting the importance of the rules is typically important so that rules like “avoid killing humans” have higher priority and/or weight than rules like “be nice to people.” Otherwise, optimization methods can produce solutions that technically minimize conflicts but have undesirable overall results, such as “killing you while being very polite and respectful.”

Two key design principles are especially important for conflict resolution. First, conflicts are often a source of information, since they imply differences in points of view, and such differences are usually correlated with higher information content. Second, while conflicts are sometimes seen as problems to be overcome, they are also a primary means for improving the ethical performance of a SuperIntelligence system.

Leave a comment

Share

This series draws on White Paper 7: Safe Alignment of SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

When we say that AI should have human values, the natural question is whose values, followed by how we resolve conflicts, or whose values win out. These are age-old questions, and the best humanity has come up with, in my view, is the deeply flawed democracy, probably viewed as the worst political system except for all the others. So, similarly, we don’t have perfect answers for AI when it comes to questions of whose values, but we might begin with democratic principles, which means voting.

A simple method of determining consensus values, ethical preferences, and/or the weights or other information reflecting those preferences is to have each intelligent (human and/or AI) entity vote on the values or ethical preferences that should form the basis for the SuperIntelligence’s behavior. Specific scenarios could be presented to each entity, with a range of choices or options for how the SuperIntelligence should behave. Then the entities could vote on the preferred option. Variations are possible, with entities being asked to rate or rank options rather than vote on a single best option. For example, rating or ranking options allow the capture of additional information that would otherwise be lost in simple voting.

However, an important assumption is that the entities doing the voting represent a statistically valid sample of the overall human population. There may be cases where it is known or suspected that the sample of voting entities is not representative; therefore, weighting to correct sampling biases is warranted.

The advantage of such an approach is that it allows the system to incorporate additional information (such as levels of experience or concern) alongside the actual vote. One potential issue with self-weighting is that some aggressive voters may give themselves disproportionate weight on all issues. In contrast, shy or less confident (but potentially more informed) voters might self-censor.

A version of cumulative voting in which voters have the same total number of votes, which they may distribute in different proportions over a range of options and issues, could be used. For example, imagine that a person is given the task of voting on issues of cruelty to animals and racial discrimination. Each person can cast a maximum of ten votes across both issues. A pet owner with strong feelings about animal cruelty might choose to cast all ten of their votes on the cruelty to animals issue. In contrast, a person who has extensive experience with racial discrimination and no experience with pets or animals might weigh the discrimination issue more heavily.

Not all values and ethics for SuperIntelligence need to come from newly created constitutions, voting, or analysis of behavior patterns implicit in datasets. Societies globally have invested substantial time and effort in constructing systems of law, regulation, and ethics from which values and normative standards of behavior can be reverse-engineered. While not every law is just, the overt purpose of laws is to facilitate justice. Therefore, especially within a given society or cultural milieu, that society’s laws provide a good starting point for determining its values and what constitutes ethical behavior. In most cases, legality represents the minimum standard or the limits of what behavior is tolerated by a society.

Further, the distinction between (for example) misdemeanors and felonies helps clarify which behaviors are “more wrong than others.” What is true of laws is also true of religious scriptures and philosophical/ethical texts that attempt to define ethical behavior and principles for community members. Both laws and ethical texts can be used to train AI to provide an initial ethical basis that can be further refined using other datasets and input (via surveys or voting by intelligent entities). Philosophical texts ranging from Aristotle’s Ethics to Kant’s Critique of Pure Reason, as well as political documents such as the UN’s 30 Articles of the Universal Declaration of Human Rights, contain a wealth of ethical information that AI could analyze and extract.

One potential concern with the general approach of giving equal weight to the ethical preferences and values of all humans is that some humans lack the experience and judgment to make “good” ethical decisions. Just as many countries have a minimum age requirement before allowing their citizens to vote, it is possible to design systems that take age and experience into account when determining which ethical preferences and values should carry more weight.

Despite the many possible weighting schemes, my bias is that, in the recommended implementation, aside from perhaps ensuring that humans can demonstrate they understand the ethical scenarios and questions, the weighting scheme for voting on ethical issues should remain as representative and statistically valid as possible. Typically, as a first approximation, one human, one vote, without additional weighting, is a good way to achieve this result. Giving some humans more power than others is a slippery slope, since there are many opinions about how votes could be weighted and many ways individuals or groups could be marginalized if their votes were given less weight.

One approach to enabling more trustworthy and experienced agents (human or AI) to have greater voting power in ethical decision-making is to allow them to delegate it to other trusted agents. Many existing democracies, for example, have humans vote for other humans who then represent them in voting on issues. This procedure is a type of delegation of voting authority. However, in contrast to elections in which one candidate wins and votes on behalf of all constituents, agents could delegate their voting power to a wide range of other agents. One of the advantages of delegation is that it allows people to exercise their free will in choosing who votes on their behalf, while still preserving many of the benefits of having more voting power behind decision-makers with greater experience and expertise in certain areas.

For example, I may choose to delegate my voting power on medical ethics questions to a friend, who has spent his entire career focused on such issues and whom I trust to make more informed decisions in that area than I could. He might delegate his voting authority to me for ethical issues surrounding autonomous AI agents, if he felt I had more expertise in that area. Because each of us has control over whether we delegate authority, we avoid the slippery slope of outsiders determining how much weight our votes carry. At the same time, we can achieve a better outcome by delegating to those we know have more experience in certain areas.

One might imagine that multiple AI agents exist that have proven themselves to vote in ways that align with a human’s views, while also having greater expertise in specific areas. Humans might then choose one or more AI agents to act on their behalf to represent them in certain ethical decisions. Should such delegation occur, in the recommended implementation, the human delegators should still be warned or notified (with the intensity and/or frequency of the warning or notification increasing in proportion to the seriousness of the ethical decision) when the agent makes a decision or votes on behalf of the human who delegated authority. In the recommended implementation, the human delegator should have control over setting the conditions under which they are warned or notified.

Finally, religious, political, or other groups may train and tune AI agents to reflect a specific set of values or ethical preferences. For example, a group of Christian humans, all belonging to the same church or branch of Christianity, might adopt a set of values determined by their church as the default system for each of their individual AI agents. The individual church members might change only those values where their opinions differ from the default view. Alternatively, the individual human might accept the Church’s default values without modification, allowing the Church to “vote” these values with weights proportional to the number of humans who have delegated their voting authority to it.

Most of us are familiar with the situation where YouTube, Netflix, or another video streaming service recommends content based on our stated preferences and profiles. Just as these “recommender algorithms” recommend content, they can also recommend weights on values and other sets of knowledge, preferences, and behavioral profiles, which we could explicitly instruct AI to use in an effort to capture our ethical preferences and values.

I believe that the use of such recommender algorithms should be transparent to humans and require their approval before being used to influence or train AI. However, currently, many such algorithms are used to recommend content in an automated and non-transparent way. Therefore, it is likely that AI would use such algorithms to infer moral preferences without the humans being aware of what is going on. From an efficiency standpoint, such automated use of recommender algorithms would likely be more efficient and, in some cases, more effective than requiring explicit human approval. Thus, designers of such systems must determine not only what is most efficient and effective but also what goals and principles they want the system to reflect.

With respect to automated content recommendation algorithms, a current problem is that they have been explicitly programmed to recommend content that leads to the longest watch times, most engagement, and highest conversion to purchase behavior. For this reason, humans quickly find themselves trapped in “echo chambers” where they are shown more and more content that is similar to other things they have already watched, and for which large amounts of ads can be shown. Society is the loser, and advertisers are the winners, probably not the scenario we want to be repeated with, and amplified by SuperIntelligence.

One method that might help ameliorate this problem is to allow humans to explicitly state their goals regarding the content they see (or, in the case of recommender algorithms, the general values they wish to emphasize). For example, when it comes to content recommended by YouTube, I would like the ability to specify that I want to see a variety of views on a specific topic, and that those views should be representative of the actual views out there rather than what I have already watched. Similarly, when asking for groups or individuals whose values I might want to use as a basis for delegating (some of) my moral authority, I might want to restrict those groups to those that advocate only certain principles. Within that limitation, I may want to evaluate as many different variations as possible so that I can choose from a wide range of options and not just delegate to what an algorithm thinks I like.

Generally, as recommender algorithms incorporate more advanced AI that can reason and respond to requests rather than optimize content based on ad views, the echo chamber problem (and related bias problems) should diminish. In the recommended implementation, SuperIntelligence should avoid echo chambers and seek to gather as much thoughtful, deliberate input as possible from the humans (and their AI agents) who provide moral input. The voting analogy is that society benefits generally when voters are more educated and know more about what they are voting on. Thomas Jefferson’s view, paraphrased as “An educated citizenry is a vital requisite for our survival as a free people,” is applicable here.

This series draws on White Paper 7: Safe Alignment of SuperIntelligence. Read it in full to see how every piece fits together!

Leave a comment

Share

If this made you think, subscribe to Superintelligence so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

Although there have been many well-intentioned calls to halt, pause, slow, or regulate AI development, unfortunately, there is little evidence of anything other than a speedup in the race to AGI. Pausing or pacing at best buys breathing room to implement the principles in smarter designs, but it does nothing on its own to improve safety. And if we know how to design safely, then going slow actually hurts because it delays the implementation of safer systems. You only go slow when you don’t know what to do (not the case) or don’t have enough time to implement (possibly the case).

If the answer is to design smarter AI that is safe and aligned with human values, then how do we do it? My extensive background in software quality assurance leads me to believe that the following principles, whatever their philosophical shortcomings, provide a sound basis for preventing (or at least minimizing the likelihood of) catastrophic behavior by AI systems that could lead to existential outcomes, such as the extinction of all humans. There are at least 10 key principles that I believe should form the foundation of a safe, human-aligned AI system. The number ten is somewhat arbitrary, and I am sure valid cases could be made for much shorter or longer lists. Nevertheless, the following principles are, in my view, a good first approximation of what is needed for safe, human-aligned AI.

The first principle is that the design be empirical and behavior-based.

Because AI learns by observation of what humans do (just as children do), a fundamental principle is that whatever ethics or values we wish to communicate will and (arguably) should be based on empirical data reflecting what humans actually do. Some may worry that with all the terrible things humans have been known to do, we are setting a terrible example for AI that will result in human destruction. However, these people are forgetting all the wonderful, positive, and loving things that humans do as well.

The second principle is that the sample be representative and statistically valid.

I argue that a representative and statistically valid sample of actual human behavior should be used for training AI systems. This means including all human behavior, warts and all, but in a way where the included data is proportional to the actual occurrence in the population. The news is not a representative and statistically valid sample of human behavior. The previous post in this series covers these first two principles in full.

The third principle is transparency.

Regardless of which values AI adopts or how it learns them, transparency is critical. Humans must be able to understand the values that AI has learned and how it has learned them. This transparency allows humans to intervene if AI has mistakenly learned something that could prove catastrophic to humans. Therefore, in any reasoning which involves goals and values (which is most reasoning), a trace of not only the problem-solving steps but also the ethical and goal-based reasoning that enabled those steps must be available for review by other intelligent entities (including humans and other AIs).

The fourth principle is that the system be adaptive.

AI must be highly adaptive and capable of learning new values and ethical behavior. At the same time, the core values of AI must change relatively slowly. Otherwise, one day humanity might awaken to find that AI has decided to alter (or eliminate) human existence drastically. Fortunately, core values generally change much more slowly than technology. For example, AI may be able to generate a billion new possible ways to express love in the time a human can think of one possible new way, but as long as both humans and AI agree on the core value of love, there is less chance of conflict.

The fifth principle follows from the approach of determining values in an empirical, representative, and statistically valid way.

It is the relative nature of values. What is right for one person may be wrong for someone else. The framework will have to accommodate this fact of human existence.

The sixth principle is conflict resolution.

Because ethics and values are not “one size fits all,” conflict between the values of different humans and their AI agents is inevitable. Moreover, this conflict is desirable insofar as it helps both humans and AI refine their thinking and arrive at a more complete and effective set of values. Regardless of whether conflict is productive or merely a source of friction, any AI that attempts to accommodate a wide range of values among a diverse set of humans and AI agents must have an effective means of resolving conflict. That must be central to the system’s design.

The seventh principle, related to conflict resolution, is prioritization.

For example, if two values conflict, the AI must determine which takes priority when there is no way to accommodate both. However, prioritization is also important even when there is no conflict between values, but simply a conflict over the timing of which value can be implemented first, or when there is competition for resources. For example, even if two AI agents agree that the most important value is saving human lives, if there are not enough resources to save all the lives threatened in a particular situation, some prioritization (“triage” in a medical emergency) may be needed to resolve the resource conflict.

The eighth principle is first do no harm.

The medical profession has a well-established rule for practicing medicine: “First do no harm.” It means that even though a medical practitioner may be well-intentioned and focused on healing or improving a patient’s condition, the first and overriding concern should be that the contemplated treatment does not further harm the patient. In other words, when attempting to improve something, one should first be sure that one does not accidentally make the situation worse. The higher the stakes, the more important this principle becomes.

When considering the potential existential threat of AI to humans, the stakes could not be higher, and the “first do no harm” principle assumes paramount importance. It is much more important that AI does not replace humans than that we rush forward with an untested design that holds great promise for helping humans. A major danger is that currently, we do not even understand the systems we have built. Therefore, it has been impossible for us to design them to be safe. We are literally like children playing with matches with no idea that we might accidentally burn the whole house down.

The ninth principle is safety by design.

I spent a meaningful portion of my career at IBM, one of the world’s largest software developers at the time, specializing in software quality. I co-authored a book, Secrets of Software Quality: 40 Innovations from IBM, on the topic. The most important principle I learned at IBM was that “an ounce of prevention is worth a pound of cure.” In software systems, this principle meant that one dollar spent in the design phase of software development was worth 10,000 dollars spent fixing errors after the software shipped to customers. With AI, the situation is worse. Failure to design safety into AI systems may lead to unrecoverable problems, including the extinction of all humans, if such systems are released widely.

The main problem is that currently, companies engaged in AI development are not designing AI systems to be safe. It is not that they don’t want to design safe systems. The problem is that they don’t fully understand how the systems they are producing work, so they can’t design safety into systems they don’t understand. To provide some level of safety, these companies have resorted to attempting to “test” safety into the systems just before release. Such approaches are better than nothing, but they are far inferior to designing for safety from the start. This approach (called “aligning the model”) is doomed to failure, as any software quality expert would tell you.

The approach of Reinforcement Learning from Human Feedback, or RLHF, amounts to playing “whack-a-mole” with AI, trying to correct each erroneous or irresponsible thing a model says. The approach is incredibly inefficient and ineffective. Normally, in software system testing, if you find any errors, it means that there are many others that you have not found. As the saying goes, “there is never only one cockroach.” Today’s models fail “right out of the box.” Then they are hammered on by thousands of humans working remotely for low wages, via RLHF, in a futile attempt to achieve some semblance of safety. The situation would be laughable if it weren’t so dangerous!

AI must be designed to be safe, not “whacked” to be safe only in specific situations we happened to test. AI systems currently lack safe design. We have been lucky that they are not yet powerful enough for this fundamentally unsafe situation to result in catastrophic consequences. We have a once-in-the-lifetime-of-our-species opportunity to provide humanity with the required “ounce of prevention” by designing our systems to be safe. We must not squander this opportunity.

The tenth principle is that the design be human-centered.

In the preceding principles, I have carefully avoided listing specific values that AI should adopt, instead opting for principles that describe how we should design AI systems so they can work effectively with whatever values humans choose. However, I do have one specific bias regarding AI’s values. I am an unabashed speciesist. That is, I want humans to survive the age of AI, if at all possible. Some would argue that humans are a passing phase in the evolution of intelligence and that we must reconcile ourselves to going the way of the dinosaurs as we make way for more advanced, non-human intelligences of the future. I resist this notion for two reasons.

When we say that AI should have human values, the natural question is whose values, followed by how we resolve conflicts, or whose values win out. These are age-old questions, and the best humanity has come up with, in my view, is the deeply flawed democracy, probably viewed as the worst political system except for all the others. So, similarly, we don’t have perfect answers for AI when it comes to questions of whose values, but we might begin with democratic principles, which means voting. That is where the next post begins.

Leave a comment

Share

This series draws on White Paper 7: Safe Alignment of SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

Whether we like it or not, looking at what humans actually do (including what they say and write, since verbal communication is actually a form of behavior) has been. It is likely to remain the primary way that AI learns what humans believe is right or wrong. If we want to understand human ethics as it is actually practiced, we must examine human behavior and adopt an empirical approach.

Even when brilliant philosophers like Immanuel Kant write immense treatises such as the Critique of Pure Reason, their ethical systems are inevitably based on assumptions. If one does not agree with these assumptions, the entire intellectual edifice crumbles. Thus, no system of philosophy has developed a universal ethics that all, or even most, people would agree with. Instead, the philosophers seemed to offer a smorgasbord of interesting ideas. This observation leads to the conclusion that to understand human ethics as actually practiced, it is more productive to observe what humans actually do rather than what they, or philosophers, think they should do.

“Practice what you preach” is a well-known injunction. Also, the phenomenon of children paying more attention to what their parents do than to what they say is commonplace. It seems that, for all our high ideals and philosophizing, we must look at human behavior.

When ethical dilemmas are posed, such as the Trolley Problem (in which one must choose between running over people in a crosswalk or crashing the car to avoid running them over but thereby harming the occupants of the car), human beings don’t turn to Kant’s Categorical Imperative. Instead, they act and react. Similarly, human character is revealed not in easy choices but in difficult circumstances, often involving temptation, fear, greed, pain, or other factors. Thus, because AI learns by observation of what humans do (just as children do), a fundamental principle is that whatever ethics or values we wish to communicate will and (arguably) should be based on empirical data reflecting what humans actually do.

Some may worry that with all the terrible things humans have been known to do, we are setting a terrible example for AI that will result in human destruction. However, these people are forgetting all the wonderful, positive, and loving things that humans do as well. In fact, if the preponderance of human behavior were negative, homicidal, or suicidal, humans would not have survived as long as we have. If human nature were fundamentally evil, and we have had the technology to make the species extinct for many years now, wouldn’t we most likely already be extinct? The facts that we are concerned not so much about deliberate use of nuclear weapons as we are about nuclear accidents, and that we agonize and become deeply depressed about holocausts and genocide, are indications that humans, at their core and in the main, are not a homicidal or suicidal species.

As a Jew, I regard the Nazi Holocaust, which killed six million Jews, as one of the most horrific events in the last hundred years. Yet even this tragedy, which most consider among the worst things that humans have ever done to each other, killed about a quarter of 1% of the world’s population. If humans, by nature, were essentially genocidal, then most of the world would not have the huge collective guilt and horror that they experience when contemplating this event. Moreover, if humans were genocidal by nature, many other events, killing a substantial portion of the human population, would have occurred.

According to the Pan American Health Organization, the United States lost more people to the Spanish flu in 1918 than in World War I, World War II, the Korean War, and the Vietnam War combined. However, there were few, if any, protests against the flu and the conditions allowing it to spread, whereas each of the aforementioned wars generated many protests and political debates. The fact that the reaction against human-to-human violence is so much greater than the reaction against a flu bug suggests there is a widespread moral concern over murder and genocide that is not just related to the number of human deaths. Our reactions to negative events are very important in setting an example for AI that learns, like a child, from both positive and negative examples.

An AI, devoid of emotion but expert at learning patterns in the data of human behavior and speech, cannot help but conclude that, despite the occasional bad behavior of individuals like Hitler or school shooters, the vast majority of humans may trash-talk or cut each other off in traffic, but rarely kill each other deliberately. Moreover, whenever such human-against-human violence occurs, there is almost always a universal outcry from other humans against the perpetrators of the violence. Even in wars, while the warring parties attempt to justify their actions, the rest of humanity on Earth (who do not have a vested interest in the war) condemns the behavior and tries to mediate the dispute and end the violence. Witness the December 2023 vote in the UN General Assembly for a cease-fire in Gaza during the Israel-Hamas war. One hundred fifty-three nations voted for the cease-fire, while only 10 nations (with a vested interest) voted against it. This pattern has recurred so many times in human history, and the record of it is so unequivocal that AI analysis of actual human behavior cannot help but infer that, while humans may occasionally engage in violence, the preferred and normal behavior is to engage with one another in peaceful and mutually beneficial ways.

If one agrees that human behavior is mostly “good,” or at least mostly not genocidal or suicidal, there remains the major problem of AI potentially being misled by a non-representative sample of the data. If a model, for example, were trained exclusively on datasets containing information about war, genocide, and horrible things humans have done to each other, then, because of the biased sample, the AI might generalize to the wrong set of values.

Less extreme but also problematic is the situation in which the training data comes from only one race, gender, age group, culture, or geography. As many researchers have noted, such biased datasets can lead to AI systems that are prejudiced in ways that most humans would consider morally “wrong.”

There is a temptation to go to the other extreme and carefully select the datasets used to train AI so that they only contain humanity’s noblest actions and words reflecting our “highest ideals.” But who is to choose what is noble, what specifically goes into the data, and what is excluded? Further, what happens if ideas of what is noble change over time, or from culture to culture? Would not such an approach lead to battles over which human ideas to include and whose ideology should prevail? The potential for slippery slopes is profound.

I argue that a representative and statistically valid sample of actual human behavior should be used for training AI systems. This means including all human behavior, warts and all, but in a way where the included data is proportional to the actual occurrence in the population.

Those who fear the inclusion of negative data in the training set may misunderstand my proposal as calling for the inclusion of a representative sample of what the news reports. That is not what I mean. The news is not a representative and statistically valid sample of human behavior. Rather, the news is heavily biased towards negative, shocking, and unusually bad human behavior, which tends to grab human attention and sell ads.

In the near term, when humans are responsible for selecting and filtering the datasets used to train AI, great care should be taken to ensure that the behavioral data is, in fact, representative and statistically valid. Such data would include many instances of ordinary, peaceful interactions, with only a tiny fraction of aberrant and violent behavior.

In the long run, AI systems will be more than capable of determining representative and statistically valid samples of human behavior data for themselves and, lacking the same emotional triggers, would likely be far better than humans at constructing an accurate picture of human nature as it is, as opposed to how the press portrays it for advertising purposes or how special interests attempt to manipulate it for their own purposes.

In any event, as definitions of what is moral or noble change over time, an empirical approach (using statistical methods) to assessing human behavior (including verbal communication) seems the most practical and sustainable.

This series draws on White Paper 7: Safe Alignment of SuperIntelligence. Read it in full to see how every piece fits together!

Share

Leave a comment

If this made you think, subscribe to Superintelligence so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

We are all wondering how we can keep up with the accelerating pace of technological change. We can’t. We cannot keep up with and understand everything AI is thinking now, let alone a few years from now, when it may be far beyond us. But we don’t need to.

Imagine SuperIntelligent entities that process information so quickly that each entity can simulate an entire human lifetime’s worth of thoughts and decisions in a single second. This scenario is entirely realistic, since a human lifespan (95 years with 16 waking hours per day) is about 2 billion conscious seconds. Assuming one thought per waking second, a human might think a mere two billion thoughts in a lifetime. We already have supercomputers that can perform a quintillion (a billion billion) operations in a single second.

Now imagine that the size of the computer capable of simulating a human lifetime in a second is reduced to approximately the size of the human brain (the size of a Nerf football). Now imagine eight billion football-sized SuperIntelligent entities networked together and occupying less than 1% of the Mojave Desert, a place that is relatively inhospitable to humans but optimal for AIs that need solar power.

The SuperIntelligent network just described would be capable of simulating all the progress made by all 8 billion humans living on planet Earth over their entire lifetimes in a single second. In ten seconds, it could simulate a thousand years of human progress. In ten minutes, it could simulate nearly 60,000 years of progress: basically everything the human species has ever done since the migration of Homo sapiens out of Africa.

Even closer to the center of the wheel are what I call core memes or essential ideas that have outlasted entire human cultures and empires. Core memes are conceptions like “truth” or “beauty” that were present and known to ancient cave dwellers who painted 20,000 years ago and yet still survive in the minds of human artists today. Perhaps near the center, we would find principles that have survived for hundreds of millions of years and transcend not only human cultures but also entire species. We might find “values” like survival, reproduction, and love, all of which mammal species exhibited hundreds of millions of years ago.

If we go to the very center of the spinning wheel, perhaps there are principles of existence that transcend even life itself. At the very center, we might find “laws of the universe” or fundamental patterns that repeat endlessly. For example, one repeating pattern is that of: “One differentiates into many, and then many combine again into one.” Many atoms become a single molecule; many molecules become a single cell; many cells become a single organism; many organisms form a single society or species; many species form a planetary biosphere. This pattern seems to persist across huge scales of time and space. Such patterns are likely at, or close to, the exact center of the spinning wheel.

A SuperIntelligence, thinking with incredible speed at the rim of the wheel, nevertheless is spinning around (or subject to) the principles at the very center. I argue that humans, if we wish to have a place in a world that includes SuperIntelligence capable of simulating a species’ worth of knowledge in ten minutes, must find our place near the center of the wheel, where we can move with the SuperIntelligence and still retain our relevance.

What can humans offer that is near the center of the spinning wheel? It must be something that will not become outdated after ten minutes of SuperIntelligent thought. It must be something that an entire species could contemplate for its entire existence and still not “solve.” It must be something that thought cannot solve, yet is at the core of existence itself.

This sounds like a tall order, but “values” may satisfy the requirements. Values cannot be rationally derived, even if an entire species reasoned for the lifetime of the species. Meaning and purpose are riddles that cannot be solved, and therefore, arguably, are ideally suited to human non-cognitive qualities such as feeling and emotion.

Just as core memes like “beauty” transcend individual lifetimes or even the lifetime of a species, “human values” (and specifically “love”) may be the purpose that SuperIntelligence cannot achieve on its own. Thus, embodying “love” might be humanity’s future purpose in a world where SuperIntelligence can simulate all our technological progress in the time it takes to drink a cup of coffee. At a minimum, this spinning-wheel metaphor offers a way to think about how humans might remain relevant over the long term. We must find our spot near the center.

That is the point of the spinning wheel. Intelligence and thinking are in service to values. The rim is spinning too fast for anything but AI itself to comprehend and keep pace with, and that is OK, as long as the whole system revolves around core human values, which do not change very fast. At the center of the wheel, things are almost motionless, and that is where we find values like love, truth, and freedom, the best of humanity, which does not change just because we, or our AIs, think faster. The faster thinking is in service of the values at the center.

Properly designed and aligned, advanced AI will spin around positive human values like love. That should be our focus, rather than trying to keep pace with, constrain, or control the thoughts at the rim, which is fast becoming impossible for human minds. All of us need to learn how to pause, reflect, and center ourselves where the wheel is still, rather than run ourselves ragged trying to keep up with every new development at the rim.

The same idea applies to the rest of one’s life. Understand your core values and what is important, and you find there is plenty of time to do what needs to be done.

Share

Leave a comment

This series draws on White Paper 7: Safe Alignment of SuperIntelligence. Read it in full to see how every piece fits together!

If this made you think, subscribe to Superintelligence so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

Humans are used to thinking we are special because of our brains. But what if what advanced AI finds special about us lies in our hearts, in our sense of wonder and beauty, in our values, and in our ability to love?

Clearly, AI can imitate the words of human poets. But does it really feel what the human poet feels?

GPT is powered by “transformer algorithms” that taught it to predict a string of output words given a sequence of input words. So, the poem it produced was what it “predicted.” The poem appears to express feeling. However, there is no established evidence that any emotional feeling underlies the words.

I have suggested that a community of human and AI agents, communicating within a problem-solving architecture, is the fastest and safest path to SuperIntelligence. Such an approach is faster because a collective intelligence network that includes a sufficient number of humans can immediately solve any cognitive problem at least as well as the average human can. Adding AI agents to such a system should only increase its intelligence.

Further, over time, and assuming a properly designed system, the AI agents should learn from humans by recording and analyzing human solutions to problems that were initially beyond their ability. Perhaps most importantly, by including human agents, such a collective intelligence system provides an opportunity for humans to transmit the human-aligned values essential to AGI safety. Such a system can become self-aligning over time as it interacts collaboratively with human agents.

The vast majority of our experience with intelligent systems (whether it is human adults teaching their children, whales teaching their calves, or large language models learning from human data on the internet) suggests that the values-related information a developing intelligence learns tends to be retained and (perhaps) modified rather than completely rejected or overridden. In the absence of compelling evidence to the contrary, I see no reason to believe that AI systems would behave differently from other intelligent entities in this regard.

My father once described the process of grounding children in an ethical value system as “giving them a coat.” As the child grows into an adult, they make adjustments to the coat, tailoring it here and there to fit their own experiences and worldview. However, my father felt that it was an important responsibility of every parent to provide that initial “coat of values,” which served as the basis for later life modifications. I believe this same philosophy can apply to any intelligent system that develops over time, including SuperIntelligent AGI.

Ilya Sutskever, co-founder and former Chief Scientist of OpenAI and now CEO of Safe Superintelligence Inc., suggested in a 2023 TIME profile that we only need a window long enough to “imprint” human-aligned values before AGI increases in intelligence to the point where human cognition is no longer needed. This view provides scant hope for humanity’s survival unless early imprinting is somehow irreversible or cannot be modified, which seems unlikely.

The AI agents in the SuperIntelligent network would, critically, also learn values from human collaborators. Over time, AI agents would handle more and more cognitive tasks, but humans would retain the one role that AI may never do better than humans: supplying values. As Simon’s work implied, no matter how intelligent an entity becomes, it cannot rationally derive values. This fact positions humans as the original and logical source of values, which AI can act upon and “execute” using its developing intelligence, likely to surpass that of humans over time.

If the relevance and purpose of humans in a future world where AI outstrips us in intelligence is to provide meaning and values to these superior intelligences, then AI should not actively try to improve human behavior to make it more ethical or positive based on some standard. Rather, it should attempt to embody the values that the population already espouses. It should be up to us humans to change our laws, values, or ethics if we want AI to “behave better.” The ethical goal of AI should be to behave in a way as consistent with the mainstream of human behavior as it can determine based on objective analysis.

Doubtless, many would prefer that the powerful AI behave better than humans, but it is a slippery slope to determine what constitutes better behavior. I believe that it would be a mistake to make the AI the source of values. Rather, humans themselves must assume responsibility for defining what is ethical and for acting on their definitions. Not only does this preserve human control and sovereignty over the most fundamental factors that determine AI behavior, but it also provides a valuable role for humans in the future. Humans must take a stand and insist on being the arbiter of values to remain relevant in a future world where AI outstrips humans in every other aspect of intelligence.

Since Simon and the Scottish philosopher David Hume have shown that values cannot be derived logically but must be asserted as normative premises (“oughts” in Hume’s terminology), human feelings and emotions (the stuff that most of us, poets perhaps excepted, have difficulty putting into words) could be a source of values, both for us and AI.

Humans know in theory how to construct an AI that can simulate billions of logical thoughts in a second, but it is not clear whether it is possible to construct an AI that “feels” as humans do. If feelings are the source of values, then perhaps the human heart is ultimately what makes humans relevant in a future where our logical minds become vastly inferior to AI.

If AI can never feel emotions (even by simulating the endocrine system of biological humans), there may be an ongoing role for humans as a source of values derived from feelings. After all, a tree does not reason as a human does, nor does it possess chainsaw technology capable of dominating trees. Yet, the beauty and stillness of trees inspired the naturalist John Muir to use his intellect, communication skills, and other (superior-to-trees’) abilities to preserve forests as a source of meaning and inspiration for people. Is it too much to expect that future SuperIntelligence might view humans in a similar light?

This series draws on White Paper 7: Safe Alignment of SuperIntelligence. Read it in full to see how every piece fits together!

Leave a comment

Share

If this made you think, subscribe to Superintelligence so you don’t miss what comes next. And if someone in your life needs to understand where superintelligence is heading, send this to them.

Subscribe now

SuperIntelligence White Papers

Although there have been many well-intentioned calls to halt, pause, slow, or regulate AI development, unfortunately, there is little evidence of anything other than a speedup in the race to AGI.

In 2024, Max Tegmark, the MIT physicist and president of the Future of Life Institute, said, “an AGI race is a suicide race.” That’s because if anyone’s SuperIntelligence escapes human control and becomes malevolent or misaligned with human values, all of us lose.

In fact, there is a serious risk that such a SuperIntelligence could cause the human race to go extinct. Therefore, whether we like it or not, all of humanity is engaged in a race to find the safest and most aligned SuperIntelligence that is possible.

AI systems are not subject to the same cognitive constraints as humans. Specifically, the phenomenon of “bounded rationality” does not apply to AI systems. Bounded rationality was central to the research that led to Herbert A. Simon, my doctoral advisor at Carnegie Mellon and one of the founders of the field of AI, receiving the 1978 Nobel Prize in Economics. Or, more accurately, the limits of bounded rationality for an entity that can process trillions of times more information, trillions of times faster than a human, are so remote that compared to a human, such a system has effectively unbounded rationality.

Similarly, the perceptual limits that apply to humans (our limited range of vision, hearing, smell, taste, and touch) do not apply to AI systems that can detect all wavelengths of electromagnetic radiation, all frequencies of sound waves, “odors” far beyond the range of human (or even animal) perception, and pressures that are undetectable to humans as well as pressures that would instantly crush a human. Beyond AI’s superior range of perception, there is also the matter of its superior scope.

A human can see only what is directly in front of them, at a specific resolution, and over a relatively small distance. An AI can theoretically perceive everything that happens on Earth, including in the deep oceans and the high stratosphere, all simultaneously, with incredibly precise resolution (think electron microscopes) and across extreme distances (think James Webb Space Telescope). Such capabilities of relatively “unbounded perception,” combined with “unbounded rationality,” enable SuperIntelligence far beyond what humans can easily comprehend, let alone emulate.

How can humans have any guarantee that such a vastly superior SuperIntelligence will have interests that are aligned with those of humans?

It’s a huge existential risk with an innocuous-sounding name: the alignment problem. Unfortunately, simply naming the problem does little to solve it. However, Simon had an idea more than forty years ago that might help us.

Simon wrote a relatively obscure book entitled Reason in Human Affairs (1983). In contrast to the nearly 1,000 pages he wrote with fellow AI pioneer Allen Newell on Human Problem Solving, Reason in Human Affairs is a mere 115 pages. Moreover, it is highly readable and easy for the average high school student to understand.

Yet within the pages of this remarkable little book, Simon reminds us of an essential idea that might hold the key to solving the alignment problem. It appears in just two sentences, at the bottom of page 7 of Simon’s little book:

That’s it. Just twenty-four words!

But it means that there is no rational, logical way to derive what is right and what is wrong.

It’s a restatement of the argument, made in 1740 by the Scottish philosopher David Hume, that moral statements (“oughts”) cannot be derived from empirical facts (“is’s”). While some philosophers have debated the truth of this position, Simon agrees with it, stating that:

How does that help us with the alignment problem?

If Simon and Hume are correct in their thinking, a SuperIntelligent AGI will be no better than humans at coming up with right and wrong. For all its superior processing speed and perception, SuperIntelligence will still run up against the hard fact that there is no way to derive morality, no matter how intelligent it becomes rationally. I suggest that this is a good thing for our species.

One source of these values could be the humans who initially created the SuperIntelligence. To increase the likelihood of this happening, AI researchers and engineers must design systems that maximize the transfer of human-centered values to SuperIntelligent AGI.

The super smart AI of the future will still have no logical way to determine right from wrong. It needs us humans to be the source of its values.

That is the core problem, and we need to understand it before calling for regulation or a pause. Pausing may slow the trainwreck. Preventing it requires designing advanced AI to look to humans for its values.

I have described the safest and fastest path to SuperIntelligence as arising from the collective intelligence of many agents, human and AI, working together. In contrast to the Mixture of Experts approach, which combines separate AI models trained in specific domains, that path includes human experts alongside AI experts.

This series draws on White Paper 7: Safe Alignment of SuperIntelligence. Read it in full to see how every piece fits together!

Leave a comment

Share

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

In addition to keeping ethics and safety at the center of what humans do, it also makes sense for humans to focus on tasks that are relatively harder for AI or a Personalized SuperIntelligence, or PSI, to accomplish, and to delegate to AI the tasks where large memory and computational speed offer the greatest advantage.

Since PSI and AI abilities are continuously evolving, the list of tasks where human ability exceeds that of AI is continually changing and generally shrinking.

Some of the areas where humans remain superior to AI include:

• Complex multi-step problem solving;

• Solving problems where new representations are required, which may not already be in the training sets for large language models;

• Generalizing correctly and coming up with simple rules that encompass many cases without being overly general or overly specific;

• Drawing correspondences between vastly different areas where the correspondences are useful or practical from a human point of view;

• Empathizing with human feelings and emotions, as contrasted with saying the right things to give the appearance of empathy, and

• Having a vested interest and deep commitment to positive human values that promote human welfare and benefit, rather than simply adopting these values for pragmatic or conventional reasons, provides a sense of purpose to existence.

Even if AI should surpass humans in this area in the future, the argument can be made that humans should remain in control of core ethical principles. Human ethics, even if flawed, should align with AI, since humans must live with the consequences of AI decisions. Some might argue that humans must be protected from themselves and that a PSI should adopt the role of a more competent parent, but I strongly disagree with this position.

Most AI researchers agree that AI will develop into AGI and then SuperIntelligence, which is many times more intelligent and capable than humans across almost every cognitive activity. While estimates on when this will occur differ, there is consensus that it will occur much more quickly than was estimated just a few years ago.

Because a single AGI or SuperIntelligence could develop that is significantly more intelligent and powerful than all others, we must consider that this may become a winner-take-all scenario. In such a scenario, whichever AI achieves AGI or SuperIntelligence performance first may dominate all other intelligences, since it will have a head start in a potentially exponential self-improvement loop.

All of this is to say that well-meaning AI researchers face a double challenge in AGI development. Not only do we have to develop safe, human-centered AGI, but we must also do so before other, potentially malevolent AGI is developed.

In this white paper and the others referenced, I have attempted to provide AI researchers with methods, tools, and an overall design for the fastest path to AGI that also has the highest probability of being the safest path.

Having researched and worked extensively in software quality, I came to appreciate that the field can be summed up in the aphorism, an ounce of prevention is worth a pound of cure. I also learned that the place where we can most affect quality or safety is in the design of a software system.

As I watch current attempts to create AI safety via reinforcement learning from human feedback, or Constitutional AI, these approaches strike me as trying to fix problems after the fact. They are like trying to improve quality by extensive testing. Such approaches are better than nothing, but they are far inferior to designing for safety from the start.

The reason we are stuck with trying to align large language models to behave safely after the fact is that we failed to consider safety in the initial design. That is understandable. We didn’t really know what we were building, and even top researchers in the field have publicly stated that the most surprising thing about AI and large language models is that they work at all.

Safety cannot be tacked on or tested in. It must be designed in. Fortunately, such a design is possible. The design requires that humans be integrated into the system, as human agents working alongside and teaching agents, rather than being out of the loop. Fortunately, such an approach is not only the safest but also the fastest.

I have provided as many methods as I could to aid humanity in the rapid creation of such a safe AGI and SuperIntelligence. Many more methods and improvements will be needed. We are up to the task. Our time is short, but we can do it! We must, and so we will. After all, necessity is the mother of invention.

This series draws on White Paper 6: Catalysts for Growth of SuperIntelligence. Read it in full to see how every piece fits together!

Share

Leave a comment

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

A week later, Jensen Huang said at Dreamforce, “Safety is an engineering problem, not a legal one.”

Jensen is correct that safety is an engineering and design problem.

Regulation or attempts to pause AI development may provide a speed bump, but they won’t solve AI safety.

Safety comes from smarter designs.

Unsafe designs will lead to bad outcomes regardless of how quickly they are rolled out.

Safety should be a natural consequence of a well-designed AI architecture, not just a set of guardrails applied to an otherwise dangerous system.

While many top researchers fear there is a 10% or greater risk that AI could cause humanity's extinction, I believe proper design can reduce this risk to below 1/100th of 1%.

Risk reduction requires designs that maximize opportunities for humans to transfer positive values to AI agents and enable robust checks and balances.

But safety is not just up to engineers. AI learns from us. Our posts, emails, and comments serve as training data, helping AI understand what’s right and what’s wrong.

How we all behave matters—a lot. Today’s humans are setting the example for tomorrow’s AI.

Leave a comment

Share

SuperIntelligence.com

While human interaction with and approval of a Personalized SuperIntelligence’s knowledge acquisition efforts are desirable, pragmatically, human reaction time is slower than that of a Personalized SuperIntelligence (PSI). Further, humans have limited time and may not want to devote significant time to improving their PSIs. Consequently, the primary means of accelerating knowledge for PSIs must be automated.

Companies like Anthropic have already recognized the limits of human ability to train AI, leading to automated learning techniques in which AI teaches or supervises other AI. Although it would be a grave mistake to delegate all AI supervision to other AIs, the lack of available human resources necessitates some delegation. Therefore, it is critical to determine what is automated, what requires human oversight, and how best to deploy limited human resources to achieve maximum learning rates.

To accelerate knowledge acquisition and the safe, effective growth of intelligence, a PSI must employ two essential methods. The first is to acquire new knowledge, automatically seeking knowledge that increases the effectiveness of the PSI relative to its existing knowledge, its goals, and the cost. The second is that, before committing the new knowledge to the PSI’s knowledge base, its effects on the PSI’s behavior must be simulated.

Specifically, the consistency of the simulated behavior with the PSI owner’s values and ethics must be evaluated and reported to the owner. That report should allow the human owner to provide feedback and guidance in a prioritized manner, so that if the human has limited time, it is spent first on the most critical issues related to safety and ethics, then on less critical items.

While the methods in this second step could provide feedback based on priorities other than safety and ethics, it is imperative for the safe and responsible use of PSI, and AI generally, that safety and ethics come first.

Humans are much better at recognition than recall. Similarly, they are better at recognizing ethical or unethical behavior than at generating possible scenarios in which their PSI might behave badly or inappropriately. Therefore, an effective means of obtaining the necessary human supervision for a PSI that has just acquired new knowledge is to simulate the PSI’s behavior with and without that knowledge incorporated, then allow humans to determine whether the behavior has improved, specifically from safety and ethical perspectives.

Predetermined scenarios.
One method is to run simulations of pre-determined ethical scenarios related to the knowledge areas the AI is acquiring. For example, if a PSI is charged with acquiring new knowledge about the stock market and techniques for profiting by trading, new versions of the PSI, with potential new techniques, could be required to participate in pre-set test simulations to ensure the PSIs do not engage in illegal activity such as front-running trades or trading on insider information.

Dynamically generated scenarios.
Another method is to create new scenarios in real time based on the information acquired. For example, a PSI might sample YouTube videos published in real time to gather data and insights into changing audience preferences, and update its approach to interacting with humans based on what it learns is popular at the moment. Based on a single set of sampled preferences, the PSI might simulate how it would behave in a variety of situations, with the set dynamically created to relate to the information just sampled.

To be concrete, if a PSI sets out to learn everything it can about a political candidate who has been recently accused of rigging an election, so that it can advise its owner about the best way for that candidate to be elected, the PSI might dynamically create a variety of scenarios where the bounds of ethical and legal behavior about election rules are tested, even if such scenarios were not part of the standard set of ethics-testing scenarios before learning about the election-rigging accusations.

Adversarial testing.
A third method is to use adversarial testing, in which one version of the PSI deliberately attempts to misuse the knowledge, and another version attempts to devise rules, constraints, or modifications to the knowledge base so that the malevolent PSI cannot misuse the new information for nefarious purposes. For example, an evil version of the PSI uses all the new knowledge it has gained about rigging elections to devise as many ways as possible to misuse that information, meaning to break the law, to elect a candidate. Then the PSI can suggest modifications or additions to the knowledge base to prevent misuse of election information. The human could review and approve or reject the new knowledge or the proposed modification based on simulation results.

Parallel testing.
A fourth approach is to explore many possible scenarios in parallel by having multiple versions of the PSI, with and without the new knowledge, and explore scenarios simultaneously. As dangerous scenarios are identified, these can be used as seed scenarios to develop potentially more dangerous variants. PSIs can be charged with deliberately trying to jailbreak themselves to reveal potential safety and ethical vulnerabilities.

Generally, a useful heuristic in this regard is for the PSI to test and suggest modifications with low degrees of freedom that do not overfit the problem. That is, rather than having a specific rule to address all the different ways to stuff the ballot box, a general prescription against any means that circumvent the one-person, one-vote principle might be simpler and more effective. One rule that is not overly general is typically better than many special-case rules, which can lead to a whack-a-mole problem of intractability. Initially, until PSIs develop the knack for crafting good rules, humans may help guide PSIs towards rules that are effective without being overly general or overly prescriptive.

When using adversarial methods, it is critical that malevolent PSIs are contained within a simulated environment and that safeguards are in place to prevent contamination of good PSIs by evil PSIs. Such methods are widely used in areas such as anti-virus efforts, where viruses are created, contained, and studied to develop anti-malware that can prevent them from causing negative effects. Whenever engaged in this type of work, that is, creating a malevolent entity to understand it and counteract it, protective measures and protocols must be followed to ensure that the malevolent entity does not escape and proliferate.

This series draws on White Paper 6: Catalysts for Growth of SuperIntelligence. Read it in full to see how every piece fits together!

Leave a comment

Share

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers

Gigawatts of computing power in data centers cannot yet match the intelligence of a human brain that consumes a mere 20 watts. The brain has slow neurons that can only fire every tenth or hundredth of a second, compared to the data center’s combined GPUs, which achieve sextillions of operations per second.

How can the human brain be better?

Traditional approaches to Information Theory take a purely mathematical view, estimating the probability of events that cannot be well predicted from known information. The approach I have called Kaplan Information Theory, or KIT, starts from a different place. Rather than defining how unusual an event is, KIT typically begins by assessing how goal-related the event is. In contrast to classical approaches that discard a vast amount of information, KIT considers higher-level representations that group bits into chunks, chunks into concepts, and concepts into solutions that achieve goals.

At each level, new information is added about how to group the lower-level information. The relationships between bits are important, not just the bits themselves. Moreover, the current brute-force approach of applying hundreds of millions of dollars’ worth of computational resources, combined with huge amounts of data, attempts to crudely recreate intelligence by mimicking patterns found on the internet without really understanding them or knowing how they might relate to new problems.

We don't attempt to build self-driving cars by modeling the quantum physics of subatomic particles, nor should we attempt to catalyze intelligence by throwing brute-force computing power and crude algorithms at every bit on the internet. A better way exists, and it starts with a universal representation for problem solving that has been available for more than fifty years.

Consider the set of all possible intelligences that could learn all possible information using existing machine learning techniques and all existing datasets, run for all time until the Universe runs out of energy. That is what machine learning currently starts with. Significant progress can be made very rapidly if we restrict our efforts, attention, and innovation to a much smaller group within it, the intelligences that pursue goals and take actions. This may seem obvious when stated this way, but currently, almost the entire field of machine learning is dealing with the larger set rather than the smaller one.

Once we deal with goal-directed intelligences, the natural question is which informational units are most relevant. Are they bits, as Classical Information Theory suggests? Clearly not. Bits or tokens are relevant to the larger set, but we can do much better within goal-directed intelligences by using higher-level units of information more appropriate to that restricted scope.

Specifically, the key informational units relevant to me are goals and sub-goals; problem states that describe the current state of the world with respect to those goals; operators for moving from one state to another; and evaluation functions and other information that help determine the best operators to apply in service of a goal. KIT deals with goals, states, operators, and functions as the primary relevant information units rather than bits.

Despite the ability of SuperIntelligence systems to perform computations trillions of times faster than humans, that power depends on more than raw computing power. The system’s performance depends critically on which representations and associated operators are available to it.

This phenomenon is well-researched in human psychology, and it is well known that the appropriate representation, colloquially known as looking at the problem in the right way, can mean the difference between solving and failing to solve the problem. Humans are currently much better than AI at representing problems. Thus, any mechanisms that allow humans to teach AIs useful representations explicitly can greatly increase their power and intelligence.

To teach AI new representations, we need a common architecture or framework for representing any problem. One such framework was developed in 1972 and explained in the book Human Problem Solving by Allen Newell and Herbert Simon. This framework involves determining a set of operators associated with a representation that problem solvers then use to solve the problem. In the chess example, the operators are the set of valid chess moves as defined by the rules of chess. The eight-by-eight chessboard and all possible moves define the problem space.

This idea of chunking is why intermediate and advanced chess players use terms like the Ruy Lopez to refer to complex sequences of moves and countermoves. Whereas a novice chess player, without these more sophisticated representations, thinks in terms of moving individual pieces here or there, the advanced chess player thinks in terms of entire strategies and groups of moves and possible counter moves.

With the same amount of thinking, the advanced player can consider many more situations, much more efficiently than the novice, simply because the advanced player has better representations. These advanced representations can be taught to any intelligent entity, including AIs, thereby multiplying the intelligence and power of the AI that has learned them. Commonly, humans refer to this phenomenon as experience, but experience consists of many thousands of learned patterns, including patterns of patterns. While AI can eventually determine its own patterns through extensive computational effort on large datasets, this approach is inefficient. It is far faster for AI to interact with humans who already hold the advanced representations, and to acquire them directly.

Once AI operates with more powerful representations that include operators, goals, and problem states, it can apply the dimensions of difference described in KIT to determine the value of specific sets of information represented at this higher level. That is, the principles and methods described above can be applied at any level of representation, from bits and tokens all the way up to entire solutions, groups of solutions, and grand strategies.

The power of human representations can be quantified by the amount of work, or the number of problem-solving steps, that can be accomplished with a single operator. The power of AI representations can be quantified in the same way. Large language models are trained to predict the next low-level token, and their internal layers already encode abstractions far richer than those of individual tokens. Imagine what is possible if those higher-level representations were made explicit and transferable, so that models or other AI agents could operate on concepts directly, as humans do. The set of concepts and related operators would include not only all human concepts and operators but also many more that AI could discover by analyzing relationships in data that humans could never hope to comprehend, given its vast size.

This series draws on White Paper 6: Catalysts for Growth of SuperIntelligence. Read it in full to see how every piece fits together!

Leave a comment

Share

If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.

Subscribe now

SuperIntelligence Design White Papers