A wasp in sharp focus on the edge of a desk; behind it, a laptop screen showing a blurred news broadcast. Below, a cream band reads: Superintelligence. The wasp looms larger. A sting is nothing next to war. The wasp wins our attention anyway. Self-Aware SuperIntelligence Series. By Dr. Craig A. Kaplan

Many people are dying as a result of wars. Yet because these facts are outside of my immediate perception, and because it is difficult to grasp what is happening cognitively, I have a relatively dim awareness of what is going on, compared to, for example, a wasp that is hovering right next to me. The suffering caused by a wasp sting pales in comparison to the death and atrocities of war. Yet it is more immediately present and looms larger in my awareness because of how my cognitive system operates.

One standard approach to defining awareness in an intelligent system is to operationalize the definition and treat it as a behavior. That is, we might be tempted to define a system as “aware” if it acts as if it is aware.

While this approach has the advantage of being practical, enabling relatively straightforward measurement of a system’s “awareness,” it is also unsatisfying. Humans know that it is possible to be aware even if there are no external signs or behaviors indicating awareness. Someone paralyzed by the drug curare and on a respirator, for example, can think, yet be unable to move, communicate, or give any indication of their state of awareness. Similarly, Stephen Hawking engaged in complex theoretical physics without any outward sign of his awareness (except when he spoke via a computer). Yet, no one would say that Dr. Hawking was unaware. So, operational or behavioral definitions of awareness capture only the subset of awareness demonstrated through behavior and miss much of what humans normally consider part of awareness.

Another approach to awareness is to consider the cognitive systems that support awareness and draw conclusions about potential awareness based on the limits of these cognitive systems. For example, without a visual sensory system (eyes) and a way of processing visual information (the visual cortex), it is difficult to imagine that an entity would have an awareness that includes vision in the same way as an entity possessing these systems. Similarly, an entity with much smaller memory and information processing capabilities is unlikely to be aware of complex representations of the world in the same way as an entity with greater memory and processing capabilities. We do not expect an ant to understand complex theories of nuclear physics, for example. Thus, perceptual, memory-related, information-processing, and other cognitive abilities place bounds on the types and scope of awareness an entity can have, regardless of its observable behavior.

Finally, both subjectively and supported by considerable experimental research in cognitive psychology, neuroscience, and related fields, attention is closely related to awareness. The general conclusion is that without attention, there is no awareness. Although there may still be unconscious cognitive activity (perceptions that are never attended to), awareness is generally limited to attended cognitive events. These observations lead to the definition of awareness used in this white paper.

Since self-awareness is a special case of general awareness, and since we have described how awareness itself depends on and is limited by the bounds of perception and rationality (or, more generally, information-processing capabilities), it follows that self-awareness and the related concept of identity are likewise limited by cognitive abilities. The implications of this fact are subtle but profound.

An intelligent entity with very limited perceptual and information-processing capabilities will be capable of far less general awareness than a more complex and capable entity. To the degree that an AI chess program could even be said to be aware, for example, it can be narrowly aware only of the game of chess. That type of awareness, devoid of a sense of self or the world other than the chess board, is so narrow and limited that most humans would consider the idea that the chess program is aware to be ludicrous. Yet, in a certain sense, the program is more aware of its limited chess world than the most brilliant humans, since it can detect patterns and reason within this very narrow field better than any human on earth. In this case, a behavioral definition of awareness would certainly lead to the conclusion that the program is aware of chess and its rules, since it demonstrates this knowledge by taking action (responding to moves and communicating its moves). But chess programs are typically not designed to be aware of anything outside of chess and often lack self-awareness or a sense of identity.

The Nobel Laureate Herbert Simon proposed a theory of “bounded rationality” that explained why humans sometimes acted irrationally. Simon suggested that the limits to their cognitive capabilities, including memory and processing limitations, led humans to “satisfice” or opt for “good enough” approximate solutions to problems that were too complex for them to solve easily. The problem of determining the optimal place to shop given the cost of gas, the traffic condition, the value of one’s time, the shelf-life of the groceries, the length of lines at different shops, the various prices of items at different shops, the sales currently underway, the coupons offered by manufacturers, etc. is simply too complex for humans to compute if they want the absolute best or optimal solution to the grocery shopping problem. However, humans can “satisfice” by simply going to a shop that usually has good enough prices on most items, and that is fairly close. That shopping decision is unlikely to be the best solution, but it is manageable, given humans’ cognitive abilities. Human behavior reflects these cognitive limits, and the result is Simon’s “bounded rationality.”

Just as humans exhibit bounded rationality, they also have bounded perception (limited by their sensory systems) and exhibit bounded awareness.

A non-human intelligent entity, such as a SuperIntelligent AI that is hooked up to satellite cameras and sensors covering the Earth and can process, in a fraction of a second, the same quantity of information that I process in my entire lifetime, obviously has the capability for far less bounded awareness. Its level of situation awareness is so much greater than mine that I might be tempted to say that my puny human-level awareness hardly counts at all, the same way I might think that the limited awareness of a bacterium hardly merits being called aware. Yet, in the case of the bacterium, myself, and the SuperIntelligence, the fundamental information-processing capabilities necessary for awareness all exist. The bacterium (if it is photophilic) responds to light and dark, swimming toward the light. Its awareness is very basic, yet it requires perceiving and processing information to produce its behavior. As a human, I have a limited awareness of war from news reports, videos, and other information I process. The SuperIntelligence would have a much more comprehensive and detailed perception of the events happening on Planet Earth, combined with much more powerful capabilities to process this information, resulting in a greater sense of awareness.

A bar chart in bits of information per second, on a scale where each step is 100 times the last. What the senses take in: about 11 million. What attention can handle: about 50. A SuperIntelligence: a bar running off the chart, labeled no fixed bound. Caption: The actual awareness of the entity is typically much smaller than the potential awareness.

The sense of self, as argued above, is a special type of awareness. Without awareness, there is no self-awareness. And the fewer perceptual and cognitive limits, the vaster and more comprehensive awareness, and thus self-awareness, can also be.

We now turn to methods for operationalizing awareness, self-awareness, and identity in non-human intelligent entities, such as AI systems. Every system can be thought of as having three levels of awareness. The most comprehensive level is potential awareness. Potential awareness encompasses all events the entity could be aware of, given the bounds and limits of its perceptual and cognitive systems.

A subset of potential awareness is current awareness. Current awareness includes the events to which the entity is directing its attention and is therefore aware of at a given point in time.

Although it is possible to have current awareness without a sense of self (when one is lost in thought and loses track of time and self, or is “lost” in the awesome beauty of a sunset), self-awareness is usually the center of current awareness. Self-awareness is the portion of current awareness that usually includes a sense of self, or identity, which serves as a central concept for unifying and making sense of perceptions and thoughts in current awareness.

From a design perspective, an entity’s perceptual and other cognitive systems and abilities define the potential and limits of its awareness. The actual awareness of the entity is typically much smaller than the potential awareness. It is limited to those events that attract the attention of the entity, or to which the entity directs its (“spotlight of”) attention. Interrupts, such as hearing one’s name mentioned at a noisy cocktail party, also form part of an entity’s current awareness.

Finally, to make sense of the world and to determine which actions to take, a sense of self-awareness or identity is helpful. In particular, for more complex intelligent entities such as humans and advanced forms of AI, a sense of identity allows the entity to act autonomously, basing cognition and other actions on how events in current awareness relate to the identified sense of self, including the self’s goals and objectives.

Adding layers of self-reflection and analysis on top of the sense of self enables the entity to modify its identity, including scaling its sense of self and identity to be larger and more encompassing, or more narrowly focused as available cognitive resources allow and as goals and objectives may dictate.

This dynamic ability to change and scale awareness, generally, and the sense of self-awareness and identity, in particular, is critical not only to the optimal functioning of intelligent entities but also to their safety (from a human perspective).

Therefore, one extremely useful aspect of this white paper is its discussion of the systems and methods that enable dynamic, scalable awareness.

The next post takes up those methods.

This series draws on White Paper 9: Self-Aware SuperIntelligence. Read it in full to see how every piece fits together!

SuperIntelligence Can Be Aware of Far More…