From: Harry Chesley (chesley@acm.org)
Date: Sat Nov 24 2007 - 13:31:19 MST
John K Clark wrote:
>
> Even if you live in a universe that was completely deterministic and
> randomness did not exist, even if you discount chaos, even if you
> ignore the influence of the outside environment, I could still write
> in 5 minutes a very short program that will behave in ways NOBODY or
> NOTHING in the known universe understands; it would simply be a
> program that looks for the first even number greater than 4 that is
> not the sum of two primes greater than 2, and when it finds that
> number it then stops. When will this program stop, will it ever stop?
> There is no way to tell, all you can do is watch it and see what it
> does, and randomness or chaos or the environment has nothing to do
> with it.
Ah, I believe I get it now: you mean that the AI is unpredictable from
our perspective. Indeed, I would hope it would be. But if I understand,
you go further and say that not only it's overall behavior, but its
behavior relative to its top-level goals is unpredictable, and hence we
cannot count on the top-level goals protecting us. I agree that this is
something worth worrying about, but may not be unavoidable.
I also believe it's possible to separate understanding from motivation.
For example, there are chess playing programs that understand chess
better than any human, but I don't worry that they will take over the
planet in an attempt to solve chess completely. And there are programs
that analyze the stock market, but don't necessarily feel a need to buy
stocks or otherwise manipulate the market. One can have a very deep
understanding of something with no desire to do anything with that
knowledge. (As an aside, I think Arthur Conan Doyle made this point with
his Mycroft character long ago.)
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:01:01 MDT