How AI Learns Values the Way Children Do
Geoffrey Hinton was right. AI is a child, and humans are its parents.
An AAAI, or Advanced Autonomous Artificial Intelligence, learns its owner’s values the same way a child learns its parents’ values. It is taught and corrected over many interactions, until the owner’s judgments become part of how it thinks.
Nobel laureate Geoffrey Hinton, who co-invented the algorithms underlying much of modern AI, has compared AI to a child and humans to its parents.
That framing aligns with the architecture of White Paper 2 (Ethical and Safe AGI) and captures something the technical literature often misses. The relationship between an AAAI and its owner is not the same as that between a tool and its user. It is closer to the relationship between a learner and a teacher who has full responsibility for what gets learned.
Every AAAI begins as a Base AI, an AI system such as a large language model trained on general data but not yet customized with user-specific information.
Examples include models in the GPT family, Google’s Gemini, Anthropic’s Claude, Meta’s Llama, DeepSeek, Apple’s Siri, Amazon’s Alexa, NVIDIA’s Nemotron series, and many others capable of understanding and responding in natural language.In the simplest scenario, the user talks to the Base AI, which responds, and through that dialogue, the Base AI determines the user’s values, goals, and objectives. It determines the ethical parameters under which the user wants to operate. It determines the types of tasks it will complete and the nature of the user’s unique knowledge, skills, expertise, wisdom, and personality.
Customization is more than question-and-answer.
Each user can teach their Base AI through questionnaire assessments, through “better or worse” comparisons that guide the system down a decision tree of variants, and by allowing it to analyze the user’s prior data. Knowledge, skills, expertise, personality, and ethical values all get customized. A travel agent’s AAAI will have deep knowledge of international logistics, airline pricing, visa requirements, and accommodation options, while a physician’s AAAI will reflect medical knowledge specific to the physician’s specialty and clinical experience.
Ethical values get customized, too.
The owner instills their values during training, and these are akin to what we call “character” in humans. We say of other people that someone is “of good character” or “trustworthy,” and such statements reflect our belief that humans have internal characteristics and values that can align with our own. During customization, each AAAI is trained on its owner’s values, learning to be good or not, based on what its owner teaches it. Several methods elicit those values: ethical scenarios generated dynamically from the user’s responses; behavior patterns drawn from partner data, such as online posts, and translated into a moral code; and ethical principles, priorities, and boundaries the user specifies directly.
One story captures the whole point.
The clearest illustration goes back to Jean, the Paris coffee expert we met in Your AI Should Think Like You. Jean had customized his AAAI with his deep knowledge of French coffee culture and his preference for cafes that source Fair Trade coffee. One day, he asked it to book a flight from San Francisco to Paris and to bring his small dog. The AAAI, which had not yet been trained on animal welfare, suggested placing the dog in the overhead bin. Jean caught the mistake and corrected it, and that correction became part of the training data for other AI agents. By capturing the ethical insight that animals require different treatment from inanimate objects, the system propagated Jean’s correction across the entire network, so that other agents who had never been trained on pet travel became more ethically calibrated because of Jean’s attention.
This is what it means for values to come from human hearts in practice. Jean did not write a policy document. He did not vote on an ethical framework. He noticed something wrong, said so, and the system learned. Multiplied across millions of users, each catching the ethical errors that arise in their actual lives, the system builds an ethical base no small group could write from a single room.
The story has a second half, because what an AAAI believes is only half of the ethical picture. There is also how it acts. To ensure ethical and efficient action in a society of AAAIs and humans, the system needs rules and norms that supplement each AAAI’s internal values, and those architectural rules are the subject of later posts in this series. The values themselves come from the people, and they enter the system one human correction at a time.
The next post takes up the mechanism that compounds those corrections at machine speed: self-play, the same technique that produced superhuman performance in chess, Go, and protein folding, now applied to value learning.
This series draws on White Paper 2: Ethical and Safe AGI. Read it in full to see how every piece fits together!
If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.