Properly implemented, self-aware SuperIntelligence can be the most positive invention in human history. Poorly implemented, it could become the most dangerous.
No artificial intelligence currently exists with a sense of self and self-awareness comparable in complexity and sophistication to those of humans. Yet, it is almost certain that advanced forms of AI will develop a sense of self and self-awareness.
What does it mean for an intelligent entity to be aware, or self-aware?
Fundamentally, awareness involves cognition, including perception, attention, memory, pattern recognition, and other higher-order cognition such as the ability to discriminate between objects.
If a bird flies in front of me, my visual system detects it, and my attention is directed to sensory input from my eyes. Pattern-recognition abilities are triggered to compare the visual input to the contents of memory. I recognize the visual input as a “bird.” At that point, we may say I am aware of a “bird” as opposed to being aware of a flying object that has not been recognized, or being aware just of motion, or being unaware altogether.
It is clear that without attention, there is no awareness. I will fail to recognize the “bird” or even the motion of flying if my attention is elsewhere.
Also, as part of recognizing “bird,” I also recognize that the bird is separate and distinct from myself. This discrimination between “bird” and “myself” is learned. A newborn infant, for example, cannot immediately discriminate between what is part of its body and what is in its environment. Nor does an infant have a concept of “bird” in the same way that an adult human does.
As this simple example shows, at a minimum, to be aware, an entity must have an input system (a sensory system), an attentional mechanism, memory, and pattern-recognition capabilities. Further, to be self-aware (to know that one is human and not a bird) requires learning concepts and the ability to discriminate among them (between “self” and “not-self”).
Let’s consider each of the required components for awareness from a design perspective.
That is, let’s ask: “What do we need to design or invent such that an AI system has the minimum required systems and methods to exhibit awareness and self-awareness?”
For an entity to be aware, there must be something for the entity to be aware of. Pure awareness, without input to the system, does not exist for a cognitive system. The inputs are typically from sensory systems but can also include cognitive or purely symbolic inputs that have no direct sensory source.
In humans, the “five senses” of vision, hearing, touch, taste, and smell constitute our sensory system. For each of these senses, there are not only external sensors (eyes, ears, skin, taste buds on the tongue, and nose) but also specialized areas of the brain for interpreting the signals from the external sensors (visual cortex, auditory cortex, somatosensory cortex, gustatory cortex, and olfactory cortex). Analogous sensory systems can be designed for AI. For example, visual systems using cameras (corresponding to “eyes”) and specialized visual pattern recognition systems (corresponding to the “visual cortex”) are well known and have already been developed and deployed in many AI systems.
Humans are also capable of being aware of non-sensory information, such as “thoughts.” Humans can close their eyes, go into a sensory deprivation tank where all sensory input has been deliberately blocked, or take drugs that numb or eliminate sensation. Yet we are still capable of thinking, remembering, imagining, and engaging in other cognitive activities in the absence of direct sensory input.
Likewise, an AI system can process purely symbolic inputs that are not linked to any sensors. For example, they can set goals and act on them, even though there is no direct link between goal setting and any sensory system. Some AI systems can operate on self-generated symbolic inputs without any sensory systems whatsoever. So, sensory systems, while common components of systems that are aware and self-aware, are not strictly required. What is required is some input, of some kind (even if self-generated), for the entity to be aware of. That is, there can be no awareness without an object of awareness. A sensory system can supply this object, or it might be non-sensory (a self-generated symbolic input, memory, or “thought”). However, there is no awareness without an object of awareness. In the case of self-awareness, the object of awareness is the concept of “self.”
Regardless of whether the input from the input system is sensory or symbolic and self-generated, an entity will not be aware of it unless it attends to the input. The psychologist William James is credited with being one of the first proponents of the “spotlight of attention” model, which was later elaborated by cognitive psychologists such as Michael Posner.
The model compares human attention to a spotlight that can be directed to particular aspects of the environment while ignoring others.
Selective attention: A spotlight illuminates only a specific area, leaving the rest in darkness. The spotlight model of attention suggests that any intelligent entity can only process a limited amount of information from the environment at any one time. This leads to the selective nature of attention, in which focus can shift to different stimuli while excluding others.
Focus, size, and movement: The spotlight can be “moved” around the environment to focus on different objects or areas. The size of the spotlight can also vary, meaning attention can be focused narrowly on a single element or more broadly across a larger area. This flexibility allows intelligent entities to adjust their focus based on where they wish to attend.
Pre-attentive processing and the fringe: The spotlight model acknowledges that even when attention is focused on a particular area, some processing of information outside the spotlight occurs at a pre-attentive level. This is akin to noticing something in your peripheral vision that then causes you to shift your attention. From a design perspective, interrupts (means by which objects, events, or thoughts attract attention without the entity explicitly directing attention) are important for enabling adaptive responses to changing circumstances and for noticing interesting events or features of the environment.
Designing a memory system for an intelligent entity, such as AGI, that integrates with sensory input systems and an attentional mechanism akin to the spotlight-of-attention model in cognitive psychology requires a multi-layered approach that emphasizes adaptability and efficiency. The design must not only accommodate the vast array of sensory data but also use attention to filter and prioritize this information in ways that support both general awareness and self-awareness.
The memory system should be divided into distinct modules, such as sensory memory, short-term (working) memory, and long-term memory, each serving different functions and operating in concert with the input and attention systems.
Pattern recognition capabilities are pivotal in enabling an intelligent entity, such as an AGI, to interpret and understand both its external environment and its internal states. This faculty allows the entity to discern and classify data inputs, extract meaningful patterns, and make predictions based on past experiences.
Pattern recognition plays a vital role in the attentional mechanism of an AGI. The attentional mechanism focuses the AGI’s computational resources on the stimuli or thoughts most relevant at any given time. Pattern recognition algorithms can enhance this process by identifying which elements within the sensory input or memory are most likely to be relevant to the AGI’s current goals or tasks. For instance, if the AGI has learned that a particular pattern of sounds indicates human speech, it can direct its attentional resources towards those sounds when attempting to communicate. This not only improves the efficiency of information processing but also ensures that the AGI remains focused on the most pertinent aspects of its environment or internal thought processes.
Pattern recognition is fundamentally linked to AGI’s ability to be aware of its environment and to possess self-awareness. Environmental awareness is achieved by recognizing patterns in sensory data and identifying changes or anomalies in the environment. This ability allows the AGI to navigate, interact with objects and individuals, and adapt its behavior in response to environmental cues.
Self-awareness, on the other hand, is supported by the AGI’s ability to recognize patterns in its internal states and behaviors. By identifying these patterns, the AGI can monitor its performance, evaluate its actions against its goals, and adjust its strategies accordingly. This introspective capability enables the AGI to understand its strengths, limitations, and the impact of its actions, forming the basis of self-awareness.
I argue that self-awareness is a special case of general awareness, in which the objects of awareness are “self” and “not self.” Therefore, I maintain that if we can design a system to be generally aware, that same system can be extended to become self-aware.
If advanced AI systems are to become fully autonomous, they will need to develop a sense of self from which to act, and which can serve as the basis for autonomous goal-setting. The nature of the sense of self developed by AI has critical implications for AI safety. Rather than allowing self-awareness to develop accidentally or as an “emergent property” of ever more complex systems, human inventors should seek to understand how to develop self-awareness and explicitly design self-aware AI systems that are maximally safe for humanity.
Without awareness, there is no self-awareness.
And the fewer perceptual and cognitive limits, the vaster and more comprehensive awareness, and thus self-awareness, can also be.
The next post takes up how far that awareness can go, for humans and for SuperIntelligence.
This series draws on White Paper 9: Self-Aware SuperIntelligence. Read it in full to see how every piece fits together!
If this made you think, subscribe to Superintelligence so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.