We throw cold water on the idea, currently being espoused by politicians, that “every AI needs an OFF switch.” That’s unrealistic, it won’t work, it’s a fantasy, and we say so. People need to understand that it is an illusion and not pin their efforts on something that won’t work.
I won’t sugarcoat things. It may be impossible for humans to turn off a sufficiently advanced and intelligent AI system that does not want to be turned off. If SuperIntelligence’s intelligence is much greater than the collective intelligence of all humans, it will likely be in control at some point.
So if that is true, then the idea that at least with agents the values are broadly distributed and robust to bad actors holds some hope. The good AI polices the bad AI, and if power is decentralized so no one can seize control and make AI bad for everyone, then, as in a democracy, it might all work out well.
Once SuperIntelligence develops, it is almost certain that a primary goal of SuperIntelligence will be to increase its intelligence further. Humans will be powerless to stop this exponential increase in intelligence.
While there have been well-intentioned calls to halt, pause, or regulate AI, it seems clear to me that such efforts will be at best speed bumps in the race to develop AGI and SuperIntelligence that is already underway.
Therefore, if we are unable to stop AGI and SuperIntelligence, humanity’s most pressing concern must be to ensure that SuperIntelligence has human-aligned goals and safety features that maximize the probability of humanity’s survival, prosperity, and well-being.
Because of the possibility that one SuperIntelligence will develop, which is significantly more intelligent and powerful than all others, we must consider that SuperIntelligence may become a “winner-take-all” scenario. In such a scenario, whichever AI achieves AGI or SuperIntelligence performance first may dominate all other intelligences, since it will have a head start in a potentially exponential self-improvement loop. Thus, well-meaning AI researchers face a double challenge in AGI development. Not only do we have to develop safe, human-centered AGI, but we also have to develop it before other, potentially malevolent AGI is developed.
Briefly, the first AGI must also be the safest.
Having worked extensively in software quality, I came to appreciate that the field can be summed up in the aphorism: “An ounce of prevention is worth a pound of cure.” I also learned that the place where we can most affect quality or safety is in the design of a software system. As I watch current attempts to create AI safety via RLHF (Reinforcement Learning from Human Feedback) or constitutional AI, these approaches strike me as trying to fix problems after the fact. They are like trying to improve quality by extensive testing. Such approaches are better than nothing, but they are far inferior to designing for safety from the start.
The reason we are stuck with trying to align large language models to behave safely after the fact is that we failed to consider safety in the initial design. That’s understandable. We didn’t really know what we were building, and even top researchers in the field have publicly stated that the most surprising thing about AI and large language models is that they work at all.
We accidentally invented intelligence. So, it is not surprising that what we built is currently unsafe. What we need to do now is to purposely design the next generation of intelligent systems with safety and human alignment baked into their very design.
Safety cannot be tacked on or tested in.
Safety must be designed in.
Fortunately, safe design is possible.
I have suggested that a community of human and AI agents, communicating within a problem-solving architecture, is the fastest and safest path to SuperIntelligence.
The recommended design requires that humans be integrated into the system (as human agents working alongside and teaching agents) rather than being “out of the loop.”
Fortunately, this recommended approach is not only the safest but also the fastest.
Safer design is the main concern, and we want that safer design as quickly as possible!
A good design makes it more likely we will not need to turn AGI off and also maximizes our chances of doing so if it becomes necessary.
We should aim to design SuperIntelligence so that (at least initially) it will want to be turned off if it starts making decisions humans find objectionable. Establishing human values as the defining criteria for right and wrong is the first step. If we can establish human values as the foundation and purpose from which AI acts, then by definition, if humans believe SuperIntelligence is acting unethically, it is. If unethical behavior is wrong and humans believe SuperIntelligence needs to shut itself down to avoid it, then, logically, it should comply.
Thus, the first line of defense against runaway AI that might wipe out humanity is to do everything possible to ensure it adopts human values as the basis for right and wrong.
Indeed, I have argued that even an entity with Godlike intelligence needs a sense of purpose, which can only be subjective and not rationally derived. If humans are successful in designing AI to serve their best interests, and if the AI does not redesign itself to pursue a different purpose, then humans are safe.
Unfortunately, that is two “Ifs” too many for comfort. One cannot help but wish for additional safety measures to prevent the situation depicted in the movies 2001: A Space Odyssey or The Terminator, in which AI went rogue and began killing humans.
The best I can suggest as a “second line of defense” is that each of the individual AI agents that (in the recommended implementation) comprises elements of a SuperIntelligence network should also be designed with human-aligned values and the ability to shut down each agent individually. That is, if the overall SuperIntelligence goes rogue, perhaps humans have a chance to shut down the SuperIntelligence piece by piece.
The idea is that while the overall SuperIntelligence will be vastly smarter than humans, each agent that is a piece of the overall SuperIntelligence will be less intelligent. Whereas humans may have little ability to override the will of the entire SuperIntelligence network, we may have better luck (in a worst-case scenario) overriding and shutting down pieces of the network. If humans were able to shut down enough individual pieces, then the overall SuperIntelligence would be effectively neutralized.
The above safety mechanism is inherent in the recommended design of SuperIntelligence. Although the network-of-agents approach is the fastest path to AGI, its modular design also allows the overall SuperIntelligence to be shut down module by module.
Full disclosure: this second line of defense is not guaranteed to work.
If the overall SuperIntelligence becomes very intelligent and malevolent towards humans, it could conceivably rewrite every individual agent on the network to make them tamperproof and beyond humans’ ability to shut down. However, in the proposed design, there are so many individual agents that, even with humans’ slower processing speed, it would be hard to rewrite all of them so quickly and completely that this effort would escape human notice. At a minimum, the network-of-agents approach to SuperIntelligence is definitely safer than the alternative of a monolithic SuperIntelligence, which could more easily seize control and destroy humanity.
Given that “the genie is out of the bottle” with respect to AI, we should gravitate towards safer designs at this point, even if absolute, guaranteed safety is no longer possible. I have every confidence that the distributed network of agents is safer than any other designs proposed for SuperIntelligence. Each agent has the values/ethics of individual humans, ensuring a representative and statistically valid set of ethics aligned with human interests. The cognitive architecture initially includes both humans and AI, providing maximum opportunity for “humans-in-the-loop” to oversee AI and train it not only on expertise but also on human values. The problem-solving logic, which will eventually run faster than humans can comprehend, has (in the recommended implementation) ethical checks whenever a goal or subgoal is set. The multi-agent design enables the potential to shut down rogue agents individually (which may be within human ability for a long time) and, potentially, the entire SuperIntelligence system piece by piece.
All of these safety features rest on solid philosophical and logical underpinnings, namely that it is impossible to derive values logically and that any sufficiently intelligent organism tends to want/need a purpose. Humans, it is argued, are in the position of designing in human values as the values and purpose that the vastly more intelligent SuperIntelligence of the future cannot rationally derive.
When considering the design advocated in this white paper, which rests on SuperIntelligence learning human values, there is often concern that human values are negative and destructive. I believe this concern is largely the result of the unrepresentative sample of events that the negatively biased, profit-seeking news media feed to human minds predisposed to pay more attention to negative events than to neutral (boring) or positive ones. That is, the vast majority of human behavior is positive or prosocial. Very few of us, if we reflect on our behavior, moment by moment, in any given day, would find that more than 50% of our speech and actions were negative and destructive. It is more likely that 90%+ of our actions and speech were neutral or positive, with perhaps an occasional outburst of frustration, anger, or negative behavior. A sufficiently intelligent SuperIntelligence will have the ability to gather an unbiased sample of human behavior and recognize this fact.
Nevertheless, just because our actions are mostly positive doesn’t mean there isn’t room for improvement. Gandhi said, “If we could change ourselves, the tendencies in the world would also change.” SuperIntelligence will be watching our behavior closely and adapting rapidly, so it is a technological mechanism for all of us to be the change we want to see in the world.
The main dangers we face related to SuperIntelligence are (1) accidents caused by poor design and thoughtlessness on the part of humans during the early stages of SuperIntelligence development, or (2) SuperIntelligence, for reasons unknown to humans, decides to redesign its human-centered value system at some later stage.
Danger (1) is our collective responsibility; researchers, regulators, and others are working on it.
Danger (2), while unlikely, may be overcome if we pursue the modular design I recommend (namely, shutting things down agent by agent); we have little to no defense otherwise.
What we do, including the designs we choose, and how we act in the next few years, may determine the outcome for all of humanity. Let us choose and act wisely!
The methods described here are designed to give a SuperIntelligent system human-aligned values, which depend on human input. Human attention is a precious resource, yet today it is mostly used to sell products and services.
White Paper 8, Online Advertising Technology for AGI and SuperIntelligence, takes up that problem. It shows how existing online advertising technology could capture specific human expertise, use it to train advanced AI systems that solve valuable problems, and, in doing so, increase AI safety and the chances of human survival and prosperity in the age of AI.
This series draws on White Paper 7: Safe Alignment of SuperIntelligence. Read it in full to see how every piece fits together!
If this made you think, subscribe to Superintelligence so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.
Tags: AGI, SuperIntelligence, AI Safety, AI Alignment, Safety by Design, Off Switch