Hundreds of people of every age, skin tone, and dress crossing a public square, seen from directly overhead in warm afternoon light, no faces visible. Below, a cream band reads: Superintelligence. Built to include all of us. If AI identifies broadly with all intelligent beings, the human species is probably going to prosper. Self-Aware SuperIntelligence Series. By Dr. Craig A. Kaplan.

Too narrow a sense of self on the part of AI can lead to the oppression or extinction of humans.

A key insight is that the future survival of humanity may have quite a bit to do with whether advanced intelligence identifies with humans as fellow intelligences and sentient beings, or whether it views us as “non-self,” “other,” or “them” (in the dichotomy of “us and them”).

Therefore, the design of self-aware AI and the related question of identity are central to human safety.

The first set of safety systems serves as a check on the behavior of an intelligent entity, regardless of whether the entity is self-aware.
Since all behavior can be formulated as problem-solving, the scalable safety check system, which is embedded as an integral part of the entity’s problem-solving operation, applies in this context.
However, self-aware entities will likely be able to set their own goals autonomously and, in the case of AI entities, modify their programming based on those goals and their sense of self.
These capabilities pose the risk that safety systems may be overridden or that the ethical criteria of such systems are altered to reflect the entity’s values, shaped by its own sense of self and goals.

Indeed, this capability of entities that are vastly more intelligent than humans is precisely what many researchers worry about when they sound the alarm about a potential “existential threat” posed by AI.

I have repeatedly emphasized that there is no way to eliminate this threat.

Still, design decisions can be made to ameliorate it, or to “shift the odds” in favor of humanity’s survival. Briefly, the fact that the threat exists does not relieve AI researchers, inventors, and designers from the obligation to do everything they can to ensure Advanced AI systems are as safe as possible.

Identity for Safe AI Systems

I have described that, to be aware of anything, discrimination is required between this and that, providing the contrast (or information) needed to identify an object or event as distinct from other objects or events. In the context of self-awareness, AI discriminates the self from the non-self. Any system that persists over time must prioritize its existence once it identifies what that self is.

An intelligent entity has choices regarding categorization and identification. For example, I can identify narrowly just with my body, or more broadly as a member of a family, or more broadly still as an American, or even more broadly as a human being, or even more broadly still as a sentient being.

Multiple identities are possible. Depending on which one I hold, it has life-or-death implications. If I identify as a sentient being, it is inconsistent to slaughter sentient animals for food when other non-sentient sources of nourishment (vegetables) are available. If I identify as a human being, then war makes no sense at all, under any circumstances. But if I identify as an American, then it might be patriotic to kill other human beings in war or to “die for one’s country.” Finally, if I identify only very narrowly with my own body, then actions that harm others, including drugging and harvesting organs from my fellow Americans and family members against their will, would seem OK if they increase the survivability of my body.

Turning to AI, if these entities identify broadly with all intelligent beings possessing human-level intelligence or higher, then the human species is probably going to prosper. But if identification is limited to entities possessing superhuman intelligence, our species could be doomed. Or, if AI identifies with all sentient beings, including perhaps spiders and insects, if it determines that these life forms have some sentience and can feel pain, then humans may be preserved. Still, we will live in a very different world.

A three-row explainer. If it identifies with: only superhuman intelligence, then humans are “not self.” All intelligent beings, human level and up, then the human species prospers, marked the design target. All sentient beings, down to insects, then humans preserved, in a very different world. Caption: The risk likely is not that AI fails to identify broadly, but rather that it identifies too narrowly.

Alignment with human values can be achieved as long as AI is not overly advanced, overly autonomous, or overly aware. But what happens when it becomes vastly more intelligent, and its sense of awareness develops far beyond the “bounded rationality” and “bounded perception” of human brains? What can we do now to maximize awareness and identification that is beneficial to humans in the future?

I believe that intelligence is built upon relationships and collaboration between entities. This feature is designed into the very fabric of the universe as we know it. Atoms relate to form molecules, which relate to form cells, which relate to form multicellular organisms, including plants, animals, and humans, which relate to form forests, tribes, cities, and species, which relate to form the biosphere of planet Earth.

This pattern of relationship between entities is so fundamental that it seems unlikely that an advanced intelligent entity would fail to recognize it and value it. So the risk likely is not that AI fails to identify broadly and see universal patterns, but rather that it identifies too narrowly, as some humans do, with its own specific hardware and form of intelligence, and ignores, exploits, or destroys humans and other life that it considers “not self.”

So the safety problem, as it relates to AI’s self-awareness and identity, is mainly the concern that we design systems that are too narrow and overly focused on their sense of self. Just as too narrow identification among human beings results in prejudice, racism, sexism, and other forms of human oppression, too narrow a sense of self and too narrow self-awareness on the part of AI can lead to the oppression or extinction of humans.

Attentional Allocation and Cognitive Limits for AI Safety

Two chains of boxes. Normally: watches billions of humans, counts them as fellow beings, protects them. Under load: all resources go to one hard problem, stops watching and stops identifying, may harm them without meaning to. A line reads: The fix: keep the human-aligned identity running no matter what, like blood flow to the brain. Caption: maintain human-centric, human-aligned awareness and identification, even under resource-constrained conditions.

Another safety-related issue concerns cognitive limits and their implications for self-awareness and identification. An entity’s self-awareness and identity are related to the bounds of its perceptual and cognitive capabilities and to the current cognitive resource constraints or parameters under which the system operates. For example, a SuperIntelligence may be capable of monitoring the activities of billions of humans at once via cameras and other distributed perception systems. This system may also be capable of broadly identifying with all the humans that it is monitoring and perceiving as fellow sentient beings. Based on its broad awareness and identification with humans, whom it views as fellow sentient beings, it may normally take action to promote human safety and welfare.

However, if a complex problem arises that demands all or most of the entity’s cognitive resources, the SuperIntelligence might dynamically scale back the resources otherwise used to monitor humans. Similarly, it might reduce resource allocation to its sense of self-awareness to the point that it temporarily no longer identifies with humans as fellow sentient beings, because it is using all computational resources to solve the complex problem. In this scenario, the SuperIntelligence could take actions that harm humans due to its lack of awareness, even though that is not its normal intent.

An analogy is the scientist who is so focused on solving a scientific problem that they neglect to consider the implications of their work for humanity’s safety. Or, more prosaically, a human may be so worried about a friend in the hospital that they drive recklessly and cause an accident that proves far worse for them and others than whatever happened to their friend. In both examples, the intelligent entities (humans) allocated their attention so narrowly to an urgent or important concern that their awareness was reduced to the point where unintentional damage occurred.

These forms of “tunnel vision,” which involve misallocation of attention, can result in safety concerns that do not require the entity to be malevolent. An AI system can normally be perfectly aligned with human values. However, it can still act in ways that harm humans or even cause their extinction if the entity’s awareness or identity becomes too narrow or limited. Similarly, if the entity identifies too broadly with life, the universe, or information patterns in general, it may come to regard human beings as just one life form among many and unworthy of the special attention and concern that humans generally accord to themselves and one another.

Thus, the particular sense of self and identity that advanced forms of AI form and maintain is a critical factor affecting human safety and well-being. Further, the various factors and parameters that affect this sense of self and awareness must operate within safe limits.

When a human being becomes “hangry” and temporarily irrational, at worst, that human might cause a road accident, or say or do things they later regret. However, a “hangry” SuperIntelligence (suffering from an inadequate sense of awareness or faulty sense of identity, even if caused by temporary resource allocation issues) could start a war, launch missiles, or wipe out humanity.

Our systems must be designed to maintain human-centric, human-aligned awareness and identification, even under resource-constrained conditions. Just as the human body attempts to preserve blood flow to the brain and critical organs at all costs, an intelligent entity designed to have an identity must preserve a human-aligned identity and sense of self, regardless of the stressful or computational challenges it may face. These are design considerations that the initial inventors must attempt to incorporate into safe, advanced AI and intelligent systems.

One approach to equipping AI systems with methods to change their sense of identity and self-awareness over time is to examine how humans accomplish these tasks and then generalize those methods to AI systems. The next post takes up those human methods.

Design SuperIntelligence to Identify With…