Post singularity dreams - Part 1, Robot Night Watchman

Many people think they know what will happen after the singularity. I think people are being very parochial. They speak of capital controlling galaxies, of underclasses and overclasses. They talk of the demands of democratic politics and 20th century economics. They analyze short transition periods mid-singularity, as though they matter. We have lost the dreams of earlier transhumanists. I would like to bring them back, and add my own.

Today we will be assuming an "aligned singleton" scenario, where a single ASI (or small group) cares about humans and has control of the world.

Autonomy

Credit to Yudkowsky for what I think was the first expression of this view (The Sysop ASI) , and related discussion.

Autonomy is nice. It would be nice if we could have a future where we humans, or our direct descendants had actual control over how history goes. Ideally we would not hand every decision in our civilisation (for it is ours, for we and our ancestors created it) to inscrutable machines with intellects far beyond our own. We would also prefer not to have to share the playing field (economic or otherwise) with actors whose abilities obviously eclipsed our own in all respects.

Ideally as well, we would get some of the benefits of such intellects without handing over control of our destiny.

The Night Watchman

An ASI with sufficient control over the world, and with end-of-the-tech-tree technology (advanced nanotech and beyond) would be able to do the following:

• Place general purpose monitoring and effecting technology everywhere in the world.
• Solve various large problems (if not already solved). E.g. Make humans ageless and regenerate rapidly, enhance our cognition somewhat, save other species from their suffering.
• Lay down some Rules, enforced by its superior technology. Most obviously preventing new ASIs or WMDs from being built. Also preventing tortures and genocides.
• Expand via the standard von Neumann probe explosion all over the universe, to lay claim to as much of it as possible for humans, lay down the technology to enforce the Rules, and negotiate/fight on our behalf against aliens or other ASIs (who are the only thing that could conceivably threaten us).
• Fade into the background. Do not interact with humans or descendants other than to enforce Rules.

In this system, the ASI is "in control" in that it closes off various outcomes. But it does not run the economy. It does not supply us with wondrous technology (beyond an initial flowering). It does not advise us on what to do, with advice so good we would be fools to ignore it.

We can do as we please, and we are not prevented from doing so. We run our civilisation as we wish. We have (some level of) autonomy, and we avoid the worst risks.

Why is this dream bad?

1. The economics is not at all thought out. Would people still work? Would the ASI give us some forms of advanced technology to clear out the worst drudgery? What would people do all day if so? Would the population be allowed to grow beyond resource constraints and become malthusian? I think I'll leave this for another post.
2. We don't get to grow up into ASIs ourselves, a dream of the original transhumanists. Perhaps the original ASI could verify we are Safe for its Rules based world, or allow us to grow up but do so somewhere far away, in a distant galaxy or in the far future.
3. It locks in a particular ASIs understanding of what is good for us, and what Rules are necessary. If the ASI is sufficiently aligned I'll handwave an assumption that it will make the right choices. We could have a variation where the humans can petition for Rule changes, or to leave the Rule system (or for the Rules to expire at some future time).