I agree with more of this than the opening led me to expect. A network of agents checking each other, values personalized by millions of people instead of written into one constitution, safety that grows with capability. That's close to how I think about it too.
The piece I'd add is where those agents run. If millions of personalized agents all live on a few providers' servers, the values are decentralized but the off-switch isn't. In my response to the Superintelligence Strategy paper, I described a framework that "begins whenever one person equips themselves with a personal agent on private compute, expands whenever two people coordinate directly as peer nodes, and gains ground simply by being genuinely useful to the participants."
You close with "We can do it if we work together!" I'd take that literally. We already have a superintelligence made of billions of black boxes keeping each other in check. It's called humanity. Machine intelligence that grows that network, rather than racing past it, is the design I'd sign up for.
I agree with more of this than the opening led me to expect. A network of agents checking each other, values personalized by millions of people instead of written into one constitution, safety that grows with capability. That's close to how I think about it too.
The piece I'd add is where those agents run. If millions of personalized agents all live on a few providers' servers, the values are decentralized but the off-switch isn't. In my response to the Superintelligence Strategy paper, I described a framework that "begins whenever one person equips themselves with a personal agent on private compute, expands whenever two people coordinate directly as peer nodes, and gains ground simply by being genuinely useful to the participants."
You close with "We can do it if we work together!" I'd take that literally. We already have a superintelligence made of billions of black boxes keeping each other in check. It's called humanity. Machine intelligence that grows that network, rather than racing past it, is the design I'd sign up for.
https://www.thesingularityproject.ai/p/superintelligence-strategy-the-perfect