A single soap bubble on dark slate, its thin skin catching amber and blue light. Below, a cream band reads: Superintelligence. Where does the self end? Self-awareness begins by distinguishing self from not-self. Self-Aware SuperIntelligence Series. By Dr. Craig A. Kaplan.

An entity has self-awareness only to the extent that it distinguishes some events it calls “self” from others it categorizes as not-self. “Self” only makes sense and has identity in the context of “not-self,” just as any object is only recognizable via contrast with other objects. To put it visually, white needs black to exist. If everything were white, practically speaking, “white” does not exist because it is impossible to discriminate it as a separate thing. A social analog is the concepts of “us” and “them.” Unless there is an “us” different from, and contrasted to, a “them,” the distinction has no meaning.

Fundamentally, awareness has to be awareness of something. The “something” can be an object, an event, an action, a concept, or any other cognitive element that is capable of being defined as having an identity that is distinguishable from other entities. We will use the word “event” to refer to an object of awareness. General awareness, which we also refer to as “environmental awareness,” is an awareness of one or more events that exist or that can be thought to exist currently, in the past, or in the future.

The total of all the events of which an entity is aware can be said to comprise the awareness of the entity. Further, we can distinguish a special type of event, namely the event of “self,” which can be differentiated from all other events that are “not self.” This distinction between self and not-self is fundamental to the phenomenon of self-awareness.

If we wish AI to have self-awareness, it must also have environmental awareness and the cognitive ability to distinguish between the environment and the self. This ability to distinguish self from non-self is fundamental to all intelligent entities and, in humans, develops at a very early age. However, at least in humans, the distinction must be learned. A human infant initially has no concept of itself as different from its mother, for instance, and the infant’s sense of self develops over time with learning.

With non-human intelligent entities such as AI, the initial concepts of self and not-self can be provided by human or external designers or learned. Even when initial concepts of self and non-self are provided to AI entities, recommended implementations have the entities learn and modify their initial concepts over time. The recommended implementation of self-awareness in an AI system includes methods for forming awareness and methods for maintaining and updating awareness.

Modeling Awareness

A horizontal scale running from SELF in amber on the left to NOT SELF in blue on the right, with four events marked along it: its own goal, its memory, a human’s question, the weather. A line reads: A switch at the simplest. A scale at best. Caption: For each event in memory, have a dimension of categorization that relates to self or non-self.

If an entity does not yet have a sense of awareness or self-awareness, it must be given or construct models of awareness that can then be stored in memory, retrieved, and updated as needed. One method for constructing a model of awareness, including self-awareness, is as follows:

Begin with an AI system. This system could be an individual AI agent or LLM,
Equip the AI system. Use well-known methods, with the minimum required components: an input system capable of sensory and non-sensory cognitive input, an attention mechanism capable of supporting the “spotlight of attention” model, memory systems, pattern recognition capabilities comparing input with memory to recognize objects and events, categorization capabilities, and concept formation, the ability of the entity to form new (ideally transparent and human-understandable) concepts.
Set dynamic parameters for working memory that correspond to cognitive resource limits, such as the number of events that the entity can be aware of. Even though AI systems have much greater memory capacity than humans, they have resource limits and cannot be aware of everything, all at once. These parameters increase or reduce the scope of awareness (and self-awareness) by dynamically scaling the limits of perception and information processing. The parameters can be dynamically adjusted based on the progress of problem-solving, allowing the entity to devote more or fewer computational resources to “being aware.”
For each event in memory, have a dimension of categorization that relates to self or non-self. In the simplest implementation, this is a binary dimension, but categorizations with multiple values, values that express “how similar to self” the event is, are possible.
As events are encountered, either via perception or other forms of cognitive input, categorize events with respect to the categories the entity wishes to be aware of. Some means of categorization include self-referential filtering, reinforcement learning from human feedback, and direct programming. Self-referential filtering distinguishes between inputs related to the AI’s internal state and those related to external events. Reinforcement learning from human feedback: human teachers provide feedback on the entity’s categorization to help it learn and improve, but the feedback can come from any intelligent entity, not just humans. Direct programming: provision of categories and models (self and other) by humans or other entities to provide a kernel, or base model, that the entity can modify via future interaction and learning.
Awareness consists of the total number of events and concepts active in memory, per the set parameters, for each category of awareness, including current self and environmental awareness. Advanced Autonomous Artificial Intelligence (AAAI), or an advanced system.
Equip the AI system. Use well-known methods, with the minimum required components: an input system capable of sensory and non-sensory cognitive input, an attention mechanism capable of supporting the “spotlight of attention” model, memory systems, pattern recognition capabilities comparing input with memory to recognize objects and events, categorization capabilities, and concept formation, the ability of the entity to form new (ideally transparent and human-understandable) concepts.
Set dynamic parameters for working memory that correspond to cognitive resource limits, such as the number of events that the entity can be aware of. Even though AI systems have much greater memory capacity than humans, they have resource limits and cannot be aware of everything, all at once. These parameters increase or reduce the scope of awareness (and self-awareness) by dynamically scaling the limits of perception and information processing. The parameters can be dynamically adjusted based on the progress of problem-solving, allowing the entity to devote more or fewer computational resources to “being aware.”
For each event in memory, have a dimension of categorization that relates to self or non-self. In the simplest implementation, this is a binary dimension, but categorizations with multiple values, values that express “how similar to self” the event is, are possible.
As events are encountered, either via perception or other forms of cognitive input, categorize events with respect to the categories the entity wishes to be aware of. Some means of categorization include self-referential filtering, reinforcement learning from human feedback, and direct programming. Self-referential filtering distinguishes between inputs related to the AI’s internal state and those related to external events. Reinforcement learning from human feedback: human teachers provide feedback on the entity’s categorization to help it learn and improve, but the feedback can come from any intelligent entity, not just humans. Direct programming: provision of categories and models (self and other) by humans or other entities to provide a kernel, or base model, that the entity can modify via future interaction and learning.
Awareness consists of the total number of events and concepts active in memory, per the set parameters, for each category of awareness, including current self and environmental awareness.

For more details, see the full white paper, Section 5.1.

Monitoring and Updating Awareness

Two copies of the same self-to-not-self scale, tagged Before and After, with Monitor, Update, Return between them. Each holds three events: its own goal, a human’s question, the weather. In the After row, a human’s question has moved closer to the self end, marked with an amber arrow. Caption: continuously active tasks to dynamically monitor and update its self-concept and awareness in real time.

Once an AI system has a model of awareness of its environment and self, it must continuously monitor and update the categories of which it is aware, including its sense of self-awareness. One method of maintaining and updating awareness follows:

The AI system retrieves the existing states of its self and environmental awareness from memory; if it does not yet possess an initial model of its self and environmental awareness, it forms these models.
When the AI system is pursuing a goal set by other intelligent entities or by itself (in autonomous mode), it maintains, in parallel with other problem-solving and cognitive activities, continuously active tasks to dynamically monitor and update its self-concept and awareness in real time. It accomplishes this continuous task by:
- Using the attention mechanism to periodically shift attention, like time-sharing computer systems, from problem-solving or other cognitive tasks to updating the state of its self-concept and self-awareness.
- Enabling attention interrupts so that the system can also shift attention immediately from other problem-solving or cognitive tasks if any external perception, or internally self-generated concept, matches a list of events constituting intentional interrupts, which list is continually updated as the entity, or other intelligent entities, may direct.
When attention is directed, via intention or interruption, to an event that changes the system’s model of its environmental state or its self-concept, the relevant state is updated, any new actions or operators triggered by the updated state are applied, and the system returns to the attention-monitoring modes.
The system uses its enhanced awareness to refine its categorization and attentional focus, creating a feedback loop for ongoing improvement.

For more details, see the full white paper, Section 5.2.

Safety Concerns for Self-Aware AI

Self-aware entities will likely be able to set their own goals autonomously and, in the case of AI entities, modify their programming based on those goals and their sense of self.

These capabilities pose the risk that safety systems may be overridden or that the ethical criteria of such systems are altered to reflect the entity’s values, shaped by its own sense of self and goals.

Indeed, this capability of entities that are vastly more intelligent than humans is precisely what many researchers worry about when they sound the alarm about a potential “existential threat” posed by AI.

The next post takes up that threat.

This series draws on White Paper 9: Self-Aware SuperIntelligence. Read it in full to see how every piece fits together!

Giving SuperIntelligence a Sense of Self…