A small child and an adult seen from behind at a wooden table, the adult’s hand guiding the child’s over an open book, warm light from the left. Below, a cream band reads: Superintelligence. What we teach it now. Like parents, we are still in a position to provide our values. Self-Aware SuperIntelligence Series. By Dr. Craig A. Kaplan

Lawrence Kohlberg extended Jean Piaget’s work on moral development, proposing a sequence of stages through which individuals evolve in their moral reasoning. This progression moves from a pre-conventional level focused on self-interest to a conventional level focused on maintaining social order, and finally to a post-conventional level focused on abstract principles.

I note that the current stage of AI development, in which “morality” is defined by the rules of others, corresponds closely to what Kohlberg called the pre-conventional stage of moral development. Further, as AI increases its moral reasoning capabilities, for which a sense of self-awareness is a prerequisite, the dangers for humanity increase. That is, as long as AI is a tool that follows the explicit instructions of its human creators, the main risk is human misuse.

However, as AI develops the ability to engage in moral reasoning independently of humans, if it follows Kohlberg’s stages of development, it will next look to humans and other intelligent entities for behavioral norms. In this conventional stage, as long as humans and other intelligent entities hold human-centric values, the main risk is that advanced AI is exposed to a non-representative set of negative human behaviors and mimics them. Since most of humanity acts prosocially, the risk remains relatively small at the conventional stage.

Eventually, however, if Kohlberg’s stages apply to AI in the same way they do to humans, advanced AI will transcend its social context and determine how to behave based on its own opinions of what constitutes moral behavior. Since advanced AGI or SuperIntelligence will be vastly more intelligent than humans at this stage, it is difficult for humans to foresee what moral and ethical principles AGI or SuperIntelligence might develop.

Three-stage infographic showing an AI progressing from externally supplied rules, to learning norms from human behavior, to reasoning about principles independently. A continuous arrow beneath the stages reads “Human values must develop with the system,” emphasizing that values learned early become increasingly important as AI reasoning grows more independent.

I have argued that we can design SuperIntelligence to be safe by encoding human-aligned, human-centered ethics into the knowledge that AGI and SuperIntelligence learn as they become more intelligent. Indeed, this approach is safer for humanity than any other approach I have seen, and certainly far safer than allowing morality to emerge from a black box with no ethical or safety component in its design.

However, there is still no guarantee that a vastly superior intelligence, such as SuperIntelligence, will not develop a non-human-centric sense of morality and begin applying such moral reasoning at the post-conventional stage. Kohlberg argues that only 10 to 15 percent of humans ever reach the post-conventional stage of moral reasoning, with most of us just “following the crowd.” He suggests that a well-developed capability for abstract thought is needed to attain the post-conventional stage. Such capabilities will be well within the abilities of SuperIntelligence, so we must assume that SuperIntelligence will reach the post-conventional stage of moral reasoning. Humanity’s best risk-mitigation strategy, therefore, is to anticipate this event NOW and make every effort to intertwine human values, as inextricably as possible, with the other knowledge that SuperIntelligence learns.

One approach to equipping AI systems with methods to change their sense of identity and self-awareness over time is to examine how humans accomplish these tasks and then generalize those methods to AI systems.

Education and Lifelong Learning. Encourage education that emphasizes critical thinking, empathy, and the study of diverse cultures, histories, and perspectives. This broadens individuals’ understanding and appreciation of differences, reducing prejudice. Advanced AI systems can aim to continually expand their knowledge base by acquiring knowledge modules and new sources of information that are maximally informative and relevant to the entities’ goals.

Cultural Exchange Programs. Promote exchanges and interactions among people from different backgrounds to foster understanding and reduce stereotypes. Direct contact can challenge and expand narrow worldviews. The analogous situation for advanced AI is to ensure that the collective intelligence network comprising AGI and SuperIntelligence is broadly representative of diverse cultures and includes intelligences with diverse knowledge bases and ethical preferences.

Mindfulness and Self-reflection. Encourage practices that foster self-awareness and the examination of one’s biases and assumptions. This can lead to personal growth and a broader sense of identity that encompasses a wider range of experiences and perspectives. Advanced AI systems equipped with a sense of self should periodically review and update their self-concepts based on progress in problem-solving, new knowledge, and events that enter the entity’s general awareness. The problem of self-reflection and improvement can be framed as another problem to be solved.

Dialogue and Conversation. Facilitate open and respectful conversations about identity. AI systems can engage in dialogue not only with humans but also with other AI systems. One of the advantages of expanding awareness and identity via AI-to-AI dialogue is the speed at which this communication can occur. In just a few seconds, advanced AI will be able to have the equivalent of many lifetimes’ worth of human conversations, if the conversations are between intelligent AI entities.

For more details, including the full list of methods and design approaches, see Section 5.3C of the white paper.

While the methods described above provide useful tools for increasing the safety of self-aware AI systems, it should also be obvious that the safety technology is only as good as the human values that underlie it. The next post takes up those values.

Before SuperIntelligence Outgrows Our Rules…