A Brief Chat on World Government

[This is the transcript of a chat conversation I had with another member of my local rationalist meet-up, on the topics of Moloch, world government, and colonization. Lightly edited for clarity, spelling, etc. and shared with their permission.]

Me: Here are some thoughts on Moloch. Moloch basically guarantees that anybody who can figure out how to successfully convert other values into economic value will out-compete the rest. So in the end, we are the paperclip maximizers, except our paperclips are dollar bills.

Scott proposes that to defeat Moloch we install a gardener, specifically a super-intelligent AI. But if you don’t think that’s going to happen, a world government seems like the next best thing. However if we escape earth before that happens, speed of light limitations will forever fragment us into competing factions impossible to garden. Therefore we should forbid any attempts to colonize Mars or other planets until we have world government and the technology to effectively manage such colonies under that government.

Them: The superorganisms in his parable only function because of… external competitive pressures. If cells didn’t need to band together to survive, they wouldn’t. If governments don’t have to fend off foreign governments they will accumulate corruption and dysfunctions.

Sort of related, I’m not persuaded by the conclusion to his parable. Won’t superintelligent AIs be subject to the same natural selective pressures as any other entity? What happens when our benevolent gardener encounters the expanding sphere of computronium from five galaxies over?

Me: Cells were surviving just fine without banding together. It was just that cells which banded together reproduced and consumed resources more effectively than those which didn’t. Similarly, I think a well constructed world government could survive just fine without competitive pressure. We haven’t necessarily found the form of that government yet, but liberal democracy seems like a decent first step.

Regarding competitive pressure on AI, he deals with that off hand by assuming that accelerating self-improvement gives an unbreakable first mover advantage. I don’t think that’s actually true, but then I’m much less bullish on super-intelligent AI in general.

Them: It would “survive,” but we don’t want a surviving government, we want a competent, benevolent one. My read on large organizations in general is that they naturally tend towards dysfunction, and it’s only competitive pressures that keep them functional.

Me: That produces a dismal view of the universe. We are given a Sophie’s Choice of either tiling the universe in economicium in order to compete and survive, or instantiating a global gardener which inherently tends towards dystopic dysfunction.

My read on large organizations in general is that they naturally tend towards dysfunction, and it’s only competitive pressures that keep them functional.

This is certainly mostly true, but I’m not yet convinced it’s necessarily true.

competitive pressures

I think this in particular is too narrow. Hunter-gatherer bands were organizations that stayed relatively “functional”, often not due to competitive pressures with other bands, but due to pure environmental survival pressures. We probably don’t want a government that stays functional due to environmental survival pressures either, but I’m generalizing to an intuition that there are other kinds of pressure.

Them: There are other kinds of pressure, but you better be damn sure you’ve got them figured out before you quash all rivals.

Me: 💯

Them: And to be precise, yeah, there’s a second thing keeping organizations intact, and that’s the floor imposed by “so incompetent they self-destruct.” But I think they degrade to the level of the floor, at which point they are no longer robust enough to survive two crises taking place at once, so they collapse anyway.

Me: Hmm, so it becomes impossible to instantiate a long-term stable gardener of any kind, and we’re stuck tiling the universe in economicium regardless.

Them: Well I think it might be possible (in the short term at least), but you have to be cognizant of the risks before you assume removing competition will make things better. So when I imagine a one-world-government, it’s more like a coordinating body above a collection of smaller states locked in fierce competition (hopefully just economic, cultural & athletic).

Me: At the risk of clarifying something which is already clear: I was never arguing that we are ready for world government now, or should work towards that soon; I was just saying there are some things we shouldn’t do until we have a good world government. We should make sure we can garden what we have before we go buying more land.

Them: Hmm, okay, I think that’s some important nuance I was overlooking.

Me: Though perhaps that is an inherently useless suggestion, since the coordination required to not buy more land is… a global gardener. Otherwise there’s competitive advantage in getting to more land first.

Them: So its a fair point. I assume that any pan-global body will not be well-designed, since it won’t be subject to competitive pressures. But its true that you might want to solve that problem before you start propagating your social structures through the universe.

Me: I’m now imagining the parallel argument playing out in Europe just post-Columbus. “We shouldn’t colonize North America until we have a well-gardened Europe”. That highlights the absurdity of it rather well.

Fast Takeoff in Biological Intelligence

[Speculative and not my area of expertise; probably wrong.]

One of the possible risks of artificial intelligence is the idea of “fast” (exponential) takeoff – that once an AI becomes even just a tiny bit smarter than humans, it will be able to recursively self-improve along an exponential curve and we’ll never be able to catch up with it, making it effectively a god in comparison to us poor humans. While human intelligence is improving over time (via natural selection and perhaps whatever causes the Flynn effect) it does so much, much more slowly and in a way that doesn’t seem to be accelerating exponentially.

But maybe gene editing changes that.

Gene editing seems about as close as a biological organism can get to recursively editing its own source code, and with recent advances (CRISPR, etc) we are plausibly much closer to functional genetic manipulation than we are to human-level AI. If this is true, humans could reach fast takeoff in our own biological intelligence well before we build an AI capable of the same thing. In this world we’re probably safe from existential AI risk; if we’re both on the same curve, it only matters who gets started first.

There are a bunch of obvious objections and weaknesses in this analogy which are worth talking through at a high level:

  • The difference between hardware and software seems relevant here. Gene editing seems more like a hardware-level capability, whereas most arguments about fast takeoff in AI talk about recursive improvement of software. It seems easy for a strong AI to recompile itself with a better algorithm, where-as it seems plausibly more difficulty for it to design and then manufacture better hardware.

    This seems like a reasonable objection, though I do have two counterpoints. The first is that, in humans at least, intelligence seems pretty closely linked to hardware. Software also seems important, but hardware puts strong upper bounds on what is possible. The second counterpoint is that our inability to effectively edit our software source code is, in some sense, a hardware problem; if we could genetically build a better human, capable of more direct meta-cognitive editing… I don’t even know what that would look like.
  • Another consideration is generation length. Even talking about hardware replacement, a recursively improving AI should be able to build a new generation on the order of weeks or months. Humans take a minimum of twelve years, and in practice quite a bit more than that most of the time. Even if we end up on the curve first, the different constant factor may dominate.
  • We don’t really understand how our own brains work. Even if we’re quite close to functional genetic editing, maybe we’re still quite far from being able to use it effectively for intelligence optimization. The AI could still effectively get there first.
  • Moloch. In a world where we do successfully reach an exponential take-off curve in our own intelligence long before AI does, Moloch could devour us all. There’s no guarantee that the editing required to make us super-intelligent wouldn’t also change or destroy our values in some fashion. We could end up with exactly the same paperclip-maximizing disaster, just executed by a biological agent with human lineage instead of by a silicon-based computer.

Given all these objections I think it’s fairly unlikely that we reach a useful biological intelligence take-off anytime soon. However if we actually are close, then the most effective spending on AI safety may not be on AI research at all – it could be on genetics and neuroscience.

Link #84 – Compressionism: A New Theory of the Mind Based on Data Compression

http://jessic.at/writing/rba.pdf

Note: not that “new” anymore, this is from 2011. Both highly technical and philosophical, but gestures towards some ideas I find interesting as far as the question of what intelligence might be.

Disclaimer: I don’t necessarily agree with or endorse everything that I link to. I link to things that are interesting and/or thought-provoking. Caveat lector.

An Incremental Solution to an Artificial Apocalypse

So smart/famous/rich people are publicly clashing over the dangers of Artificial Intelligence again. It’s happened before. Depending on who you talk to, there are several explanations of why that apocalypse might actually happen. However, as somebody with a degree in computer science and a bent for philosophy of mind, I’m not exactly worried.

The Setup

Loosely speaking, there are two different kinds of AI: “weak” and “strong”. Weak AI is the kind we have right now; if you have a modern smartphone you’re carrying around a weak AI in your pocket. This technology uses AI strategies like reinforcement learning and artificial neural networks to solve simple singular problems: speech recognition, or face recognition, or translation, or (hopefully soon) self-driving cars. Perhaps the most recent publicized success of this kind of AI was when IBM’s Watson managed to win the Jeopardy! game show.

Strong AI, on the other hand, is still strictly hypothetical. It’s the realm of sci-fi, where a robot or computer program “wakes up” one day and is suddenly conscious, probably capable of passing the Turing test. While weak AIs are clearly still machines, a strong AI would basically be a person instead, just one that happens to be really good at math.

While the routes to apocalypse tend to vary quite a bit, there are really only two end states depending on whether you worry about weak AI or strong AI.

Skynet

If you go in for strong AI, then your apocalypse of choice basically ends up looking like Skynet. In this scenario, a strong AI wakes up, decides that it’s better off without humanity for some reason (potentially related to its own self-preservation or desires), and proceeds to exterminate us. This one is pretty easy for most people to grasp.

Paperclips

If you worry more about weak AI then your apocalypse looks like paperclips, because that’s the canonical thought experiment for this case. In this scenario, a weak AI built for producing paperclips learns enough about the universe to understand that it can, in fact, produce more paperclips if it exterminates humanity and uses our corpses for raw materials.

In both scenarios, the threat is predicated upon the idea of an intelligence explosion. Because AIs (weak or strong) run on digital computers, they can do math really inconceivably fast, and thus can also work at speed in a host of other disciplines which boil down to math pretty easily: physics, electrical engineering, computer science, etc.

This means that once our apocalyptic AI gets started, it will be able to redesign its own software and hardware in order to better achieve its goals. Since this task is one to which it is ideally suited, it will quickly surpass anything a human could possibly design and achieve god-like intelligence. Game over humanity.

On Waking Up

Strong AI has been “just around the corner” since Alan Turing’s time, but in that time nobody has ever created anything remotely close to a real conscious computer. There are two potential explanations for this:

First, perhaps consciousness is magic. Whether you believe in a traditional soul or something weird like a quantum mind, maybe consciousness depends on something more than the arrangement of atoms. In this world, current AI research is barking up entirely the wrong tree and is never going to produce anything remotely like Skynet. The nature of consciousness is entirely unknown, so we can’t know if the idea of something like an intelligence explosion is even a coherent concept yet.

Alternatively, perhaps consciousness is complicated. If there’s no actual magic involved then the other reasonable alternative is that consciousness is incredibly complex. In this world, strong AI is almost certainly impossible with current hardware: even the best modern super-computer has only a tiny fraction of the power needed to run a simplistic rat brain let alone a human one. Even in a world where Moore’s law continues unabated, human-brain-level computer hardware is still centuries away, not decades. We may be only a decade away from having as many transistors as there are neurons, but they’re not comparable; neurons are massively more complex building blocks.

It’s also worth noting that if consciousness is hard because it’s complicated, our hypothetical strong AI would wake up very, very slowly. We’re not going to suddenly jump from current processors to AI-capable ones that are millions of times more powerful; there will be numerous incremental designs in between. As our processing power improves, our AI will get stronger and stronger but there will be no eureka moment where Skynet both suddenly decides humanity has to go and is smart enough to make it happen. We’ll have plenty of opportunity to study primitive artificial consciousnesses, learn about them in an environment we control, and iteratively incorporate our findings into subsequent generations.

Regardless of why we’ve never gotten anywhere with strong AI, we haven’t, and there is no indication that this time it really is “just around the corner”.

On Paperclips

While the idea of Skynet is common in popular culture, it’s the paperclip maximizer which tends to worry the people who actually seriously worry about this stuff. It’s already true that we don’t really understand how weak AI actually works: we understand the principles which drive something like Watson, but no human could describe the series of specific decisions it performs to arrive at a given answer.

It also seems a lot more plausible that weak AI (which we already have) will just keep getting smarter (which it has definitely done so far) than that we’ll have some weird magic breakthrough in strong AI, when nobody still even understands what that would look like. So, why couldn’t some future self-driving car learn just enough about the world to figure out that it would be able to drive much more efficiently if it could just get rid of all of those pesky pedestrians? As it turns out, I think something like this could actually happen; it’s just not the end of the world when it does.

As with the consciousness-is-complicated explanation for strong AI, our saving grace against paperclips and homicidal cars is the incremental nature of the technology. The first self-driving car smart enough to decide to murder pedestrians is not also going to be smart enough to redesign its own hardware, create biological weapons, and do all of this in secret without hinting to its owners that it “has plans”. No, the only thing it’s going to know how to do to achieve this goal is to try and intentionally run people over, which is obviously bad, but not exactly apocalyptic. It also has the benefit of being really freaking obvious.

Here’s a hint to future AI developers: maybe you should stop making it smarter when your cars start trying to run you over in the name of optimizing the number of pedestrians.