Customizing a Personalized SuperIntelligence, or PSI, does not have to start from scratch.
Let’s say David is a professor of theology and ethics, and he has already tuned and customized Meta’s open-source Llama 4. His version has a unique set of ethics and values, based on interactions with David, that is much more detailed and sophisticated than the base-level model. I trust David’s ethics and the customization work he has done, so I prefer to begin, with his permission, from his version rather than from the out-of-the-box model. I am particularly interested in my customized AI playing chess in a style similar to mine, but with the knowledge of chess champions Garry Kasparov and Magnus Carlsen. So the training data also includes every game I have played online, purchased datasets containing the complete games of both champions, and transcripts from chess commentators who have covered their play.
Initially, I train David’s LLM on the new datasets, assigning equal weight to each. However, I feel that the resulting LLM plays chess too much in Garry Kasparov's style and not enough in Magnus Carlsen's or mine. Using an interface with dials and sliders, I reduce the weight of Kasparov’s datasets, slightly increase the weight of Carlsen’s datasets, and increase the weights of the datasets reflecting my own chess games even more. I iteratively adjust the weights across various datasets until I am happy with the LLM’s resulting behavior.
I am not limited to adjusting the weights by hand. I can also tell an AI agent, specialized in helping humans train their agents by adjusting weights on datasets, what my desired changes are, and then let it specify exactly how to implement them. For example, I can tell the AI agent that I want it to be more aggressive in the opening and middle of the chess game, and not try to win by trading pieces and waiting for a piece advantage in the endgame.
The agent then analyzes the available chess training sets and gives more weight to games won through aggressive moves in the opening and middle of the game. I do not have to be aware of the details of this analysis or the specific changes to weight settings that the AI agent determines. Instead, I examine how the resulting version plays and provide feedback, indicating whether the result is closer to or farther from the desired chess style. After several iterations, I am satisfied with how David’s LLM now plays chess in my style.
The same process can be applied to values.
Next, I move on to ethical scenarios and, through a series of interactive dialogues with the AI training assistant, specify how I might differ in them. Although I generally share David’s ethical sensibilities, there are a few cases where David would turn the other cheek, and I believe the behavior should be more of an eye for an eye. I also specify that I do not want the eye-for-an-eye principle to extend to making the whole world blind.
I ask the AI training assistant to incorporate knowledge and research from game theory, which suggests that tit for tat ethical behavior yields the most stable and fair interactions between intelligent agents with differing objectives. At the same time, I specify that there are limits to tit for tat and that any behavior that would result in widespread destruction or loss of human life is off-limits, regardless of the behavior of the other agent. Instead, in these cases, means of neutralizing the offending party’s behavior without retaliation must be sought.
Then I spend a stint in the metaverse playing ethical games, in which the AI agent observes not only what I say, but also what I do in various situations. After that, the agent has enough information to adjust ethical training weights and present me with a series of differently customized versions of David’s LLM, from which I choose the one closest to what I had in mind.
The next post turns to what a PSI can do once it knows your knowledge and your ethics, including trading its knowledge with other agents, selling what only you know, and working while you are on vacation.
This series draws on White Paper 5: Safe Personalized SuperIntelligence. Read it in full to see how every piece fits together!
If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.