The Most Important Job AI Still Can't Do
Humans frame the problem, AI explores it, and the framing is where human values enter the system.
AI is remarkably good at solving problems. The hard part is deciding which problem to solve in the first place. Enormous effort goes into asking whether AI answers align with human values, and much less into the question that comes before it: who decided what problem the AI was solving. Alignment does not begin when an AI produces an answer. It begins earlier, when someone decides what success looks like.
Before any solution can exist, the problem has to be represented.
Someone decides the goal, defines the starting point, determines what counts as success, and identifies which constraints matter. Researchers call this problem representation (aka “framing”), and for decades it has been one of the hardest challenges in AI. Once a problem is well framed, machines are remarkably good at searching through possible solutions. Framing it in the first place is where they have struggled.
The human advantage here comes down to perception.
In the 1980s, the Nobel laureate Herbert Simon and his coauthor Jill Larkin published a famous paper on why a picture is worth a thousand words. They distinguished between having the same information and being able to use it efficiently. It might be possible to describe every aspect of a picture using words and logic, but a graphical representation is far more efficient. That is, you can see things at a glance that would take a long time to describe in words. Humans build these efficient representations naturally, using sight, hearing, touch, and a lifetime of experience in the world. We routinely take vague, messy situations and turn them into problems that can actually be solved, often without realizing we are doing it.
That difference suggests a natural division of labor.
Humans define the problem, and AI solvers rapidly explore many solution paths and present the promising ones. Of course, humans monitor, review, correct, and guide the AIs in an interactive process. Over time, the AIs acquire enough knowledge of how humans typically represent and solve problems to perform more of these tasks independently.
However, the human advantage is unlikely to last.
For years, a large language model had to understand the world solely through words, so humans held the representational edge. Today’s AIs are rapidly becoming multimodal, taking in images, audio, and video alongside language, and they already hold a huge advantage in memory and processing speed. The representational gap is closing. To the degree that humans can still contribute the formative representations, framed by human values, we should do so.
The division of labor is a safety feature, not just a practical workflow.
Whoever represents a problem determines what the system is trying to accomplish. The goals, the assumptions, and the constraints all enter at that stage. When those choices reflect human judgment, human values become part of the system before an AI ever begins its search. The values are not added afterward. They are built into the first step of the work. After all, a solution can only be as aligned as the goal it serves.
As AIs grow more capable of representing problems on their own, the uniquely human role does not disappear. Search gets faster, and memory gets larger, yet none of those advances answer the question of what is worth pursuing. That judgment should remain human, and the architecture keeps humans in the seat where it is made. Safety by design, from the first step!
This idea scales far beyond a single interaction with an AI. Almost every intellectual task can be represented as a problem. Writing a report, designing a bridge, discovering a drug, and planning a city all begin with defining the problem correctly. In our architecture, these problems form a single, continuously growing tree (aka the “WorldThink tree”), with large problems decomposed into smaller subtrees. The representation step repeats at every level, and humans can enter wherever judgment or values need to shape the search.
One difficult question remains, and it deserves a hard look. If human values guide the system, the system’s safety depends on whose values those are, and people disagree about which goals should be pursued. The next post takes up that problem directly and shows why drawing on the values of many people produces a system that is safer and more representative than one built around the values of a few.
This series draws on White Paper 3: Human-Centered AGI. Read it in full to see how every piece fits together!
If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.