Who Should Have the Most Influence on AI?
Two methods can combine the values of many, but the harder question is how much each voice should count.
Imagine millions of individuals, each customizing a personal AI agent and teaching it their knowledge and their values.
Those humans and their agents together shape the AIs that come after them, so the ethics of any new AI become the property of a whole community.
Turning that community into a single trained student is a practical matter of mechanics, namely, how the lessons of millions of teachers become the values of one AI.
There are two roads, and either one can get there.
The first road runs through feedback.
An LLM being trained adjusts its network weights based on the responses it receives, much as it does during Reinforcement Learning with Human Feedback. The difference is that the feedback can come from both humans and AI agents. This is the Reinforcement Learning via Feedback, or RLF, introduced in the previous post. Because AI agents can provide feedback much faster and at far lower cost than humans, a large community of customized agents could evaluate the same scenario together and help train a new AI.
The second road proposes combining weights directly.
Imagine two copies of the same LLM, one customized through interactions with one person and the other customized by someone else. Training has changed each model’s network weights, leaving two distinct models that carry the influence of two different people. The weights of those models could then be combined mathematically to create a third model that reflects input from both teachers. In the simplest proposed scheme, averaging the weights would give the two teachers equal influence, whereas other combinations could give one teacher more influence than the other.
For the purpose of comparing weighting schemes, the two roads are functionally equivalent.
Many teachers can shape a single student through feedback, or separately trained models can contribute through direct weight combination, and the choice of path may depend on factors such as the availability of teachers and the computational resources at hand. Either road leads to the more important question of how much each teacher should count.
The natural starting point is one agent, one vote.
Each participating human or customized AI receives equal influence over the student, and only one copy of each customized AI participates, so every teacher counts exactly once. However, equal influence is only one possible choice.
A first variation gives different weights to human and AI input.
One might expect humans to count more because people are usually better equipped to represent their own values than the AI agents they trained, yet there can be exceptions. Imagine that someone spends years teaching a personal AI agent their values and preferences, and later develops a serious cognitive impairment. The agent might then represent the person’s long-established values more faithfully than the person can express them, and in such a case, it might make sense for the agent’s input to carry more weight.
That exception must not become a reason for AI to dismiss human input altogether. Human values remain fundamental, even when AI can find better, faster, and more effective ways to pursue human goals. Later in this series, we return to the question of why greater intelligence does not give AI the authority to determine humanity’s values.
Another possible scheme gives more weight to expertise.
Physicians who have spent their careers advising terminally ill patients may have more insight into end-of-life decisions than someone who has never faced them. The design does not insist that experts should always count more; it allows input to be weighted by expertise and by its relevance to the decision being made. People are often reluctant to surrender important personal decisions to professional experts, and prevailing values preserve the “right” of people to do stupid things, as long as they are not harming others.
Expertise is also different from trust. A patient deciding whether to have an operation might seek advice from a highly knowledgeable surgeon. If that surgeon has a reputation for recommending unnecessary procedures, the patient might place greater trust in a family friend with good judgment and the patient’s best interests at heart. Of course, an expert who is also highly trusted might deserve the greatest influence of all. AI can weigh its teachers in similar ways.
Each human or AI agent can carry metadata describing relevant expertise, trustworthiness, background, preferences, and past performance, and that metadata can adjust the influence given to the agent’s input. The same approach can apply beyond values, helping to combine knowledge, skills, and judgment from many different sources.
Time can affect weighting as well. Recent input may better reflect current circumstances, and ethical norms, institutional policies, and laws change. In 2020, Disney placed warnings before some older films stating that they contained negative depictions or mistreatment of people or cultures, and that those stereotypes were wrong. By 2025, the company was using shorter language, stating that the program was presented as originally created and might contain negative depictions. The revisions show that institutions can change how they characterize the same content within a few years.
Sometimes change is gradual, and older input can slowly fade in influence, while at other times change is immediate.
During Prohibition, selling alcohol was illegal, and when Prohibition ended, its legal status changed overnight. A system that uses law as one signal of prevailing norms would need a stepwise adjustment, since no gradual fade can capture a change like that.
These weighting schemes explain how the values of many teachers can be pooled. They do not tell us what to do when sincere and thoughtful people give different answers, and no objectively correct answer exists. Researchers have documented how differently people around the world resolve one famous dilemma about who should be spared and who should be sacrificed.
My next post takes up that dilemma, the trolley problem, and shows why disagreement strengthens the case for teaching AI the values of many people instead of a chosen few.
This series draws on White Paper 4: Safe, Scalable Artificial General Intelligence. Read it in full to see how every piece fits together!
If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.