Penrose’s Mistake

Uri Sharir

Computability Seminar – Prof. Nachum DershowitzTel Aviv University

December 11, 2013

2

Outline

• Brief history of AI• Lucas and Gödel• Penrose and Turing• Summary

3

What is AI?

artificial intelligence n 1 :  a branch of computer science dealing with the simulation of intelligent behavior in computers 2 :  the capability of a machine to imitate intelligent human behavior

Merriam-Webster

4

Two Kinds of AI “Strong AI”

An appropriately programmed computer is a mind

The human mind is in reality some kind of physical symbol-manipulation system

“Weak AI” Physical symbol systems have the necessary

and sufficient means for intelligent action Allows for the possibility that the mind is not

mechanical, but claims that it can (theoretically, at least) be simulated by a Turing machine

5History: Myths and Antiquity Throughout human history,

people have used technology to emulate themselves

Greece: Hephaestus’ golden robots,

Talos, Pandora Pygmalion’s Galatea 384-322 BC: Aristotle

describes the syllogism 1st century AD: Heron of

Alexandria’s automata China:

King Mu of Zhou and Yan Shi’s “artificial actor”

6

History: Alchemy, Fiction and Hoaxes ~800: Jabir ibn Hayyan’s

Takwin 1206: Al-Jazari’s

programmable automata ~1500: Paracelsus’

homunculus ~1580: Rabbi Judah Loew ben

Bezalel of Prague’s Golem 1769: Wolfgang von

Kempelen’s The Turk 1818: Mary Shelley’s

Frankenstein 1920: Karel Čapek’s Rossum’s

Universal Robots (R.U.R.)

7

History: Formal Reasoning

1275: Ramon Llull’s Ars Magna

Early 17th century: René Descartes: bodies of animals are nothing more than complex machines (but not mind or soul)

1641: Thomas Hobbes in Leviathan: “reason is nothing but reckoning”

1672: Gottfried Leibniz: Stepped Reckoner Characteristica universalis

8

History: Formal Reasoning 1854: George Boole’s Boolean algebra

“Investigate the fundamental laws of those operations of the mind by which reasoning is performed, to give expression to them in the symbolic language of a calculus”

1913: Bertrand Russell and Alfred North Whitehead publish Principia Mathematica

1920s and 1930s: David Hilbert’s program Alonzo Church’s lambda calculus The Turing machine Kurt Gödel’s incompleteness theorems

Within certain limits, any mathematicalreasoning could be mechanized

9

History: The Birth of AI 1943:

Norbert Wiener's Cybernetics Warren Sturgis McCulloch and

Walter Pitts publish “A Logical Calculus of the Ideas Immanent in Nervous Activity”

1950: Alan Turing proposes the Turing

Test Isaac Asimov published his

Three Laws of Robotics 1951: first working AI programs 1952-1962: Arthur Samuel’s

checkers programs

10

History: The Birth of AI 1955: Allen Newell and Herbert A. Simon

created the “Logic Theorist”, eventuallyproving 38 of the first 52 theorems inPrincipia Mathematica (and finding newand more elegant proofs for some) Simon: they had “solved the venerable mind/body

problem, explaining how a system composed of matter can have the properties of mind”

Later John Searle will call this the “Strong AI Hypothesis” 1956: The first Dartmouth College summer AI

conference is organized by John McCarthy, Marvin Minsky,Nathan Rochester (IBM) andClaude Shannon

11

History: The Golden Years (1956-1974) Several fields of work:

Reasoning as search▪ Stanford’s Shakey

Natural language▪ Joseph Weizenbaum’s ELIZA

Micro-worlds▪ Terry Winograd’s SHRDLU

Optimism: 1958: “Within ten years a digital computer will be the world's chess champion”;

“within ten years a digital computer will discover and prove an important new mathematical theorem” (H.A. Simon and Allen Newell)

1965: “Machines will be capable, within twenty years, of doing any work a man can do” (H.A. Simon)

1967: “Within a generation ... the problem of creating ‘artificial intelligence’ will substantially be solved” (Marvin Minsky)

1970: “In from three to eight years we will have a machine with the general intelligence of an average human being” (Marvin Minsky)

Funding: 1963: MIT received a $2.2M grant from the newly created Advanced Research

Projects Agency (later DARPA)

A*<P,O,I,G>

12

History: The 1st AI Winter (1974-1980) Expectations set too high… Limited computing power (AI requires computer

power in the same way that aircraft require horsepower)

Intractability and combinatorial explosion (search or scaling toy micro-worlds)

Common sense and reasoning (general knowledge too vast)

Moravec’s paradox: proving theorems and solving geometry problems is comparatively easy for computers, but a supposedly simple task like recognizing a face or crossing a room without bumping into anything is extremely difficult

The end of funding…

13

Critiques from Across the Campus 1961: John Lucas in “Minds, Machines and

Gödel”: Turing’s and (more so) Gödel’s theorems prove

that “Mechanism is false”: minds cannot be explained as machines

1965, 1972: Hubert Dreyfus in Alchemy and AI and What Computers Can't Do: Challenges the four philosophical assumptions

AI researchers treated as axioms:▪ Biological (brains are composed of on/off switches)▪ Psychological (minds operate according to formal rules)▪ Epistemological (all knowledge can be formalized)▪ Ontological (the world consists of independent facts)

1980: John Searle’s Chinese room argument

14

Outline

• Brief history of AI• Lucas and Gödel• Penrose and Turing• Summary

15

John Randolph Lucas (1929-) Follows Gödel’s proof:

Assume a consistent formal system F, which is strong enough to produce simple arithmetic (contains ℕ, + and ×)

Consider the Epimenides-inspired formulag = “This formula is unprovable in F”

If g is provable in F, we get a contradiction:▪ g is false in F (by its definition)▪ g is true in F (since F is sound)

Then g must be unprovable in F, and thus g is true So all interesting consistent formal systems are incomplete:

“contain unprovable, though perfectly meaningful, formulae … we, standing outside the system, can see to be true”

In particular, this “must apply to cybernetical machines” “It follows that no machine can be a complete or adequate

model of the mind, that minds are essentially different from machines”

16

Lucas’ Argument in Other Words Assume that for any human H there exists at least one

(deterministic) formal logical system F(H) which reliably predicts H’s actions in all circumstances

For any logical system F, a sufficiently skilled mathematical logician (equipped with a sufficiently powerful computer if necessary) can construct some statements G(F) which are true but unprovable in F (Gödel)

If a human M is such a logician, then if M is given F(M), he or she can construct G(F(M)) and determine that they are true (which F(M) can’t)

Hence F(M) does not reliably predict M’s actions in all circumstances Lucas goes further to state the anti-deterministic view that this

implies that M has free will And it is implausible that the qualitative difference between

mathematical logicians and the rest of the population is such that the former have free will and the latter do not

17

More from Lucas “We are trying to produce a model of the mind which is

mechanical – which is essentially ‘dead’ – but the mind, being in fact ‘alive’, can always go one better”

“Thanks to Gödel's theorem, the mind always has the last word” Adding to the system additional axioms or rules to prove its

Gödel formula wouldn’t help, because the “strengthened” system would have a Gödel formula of its own

It’s like a game between the mechanist (M) and Lucas (L): M goes first: he produces a definite mechanical model of

the mind L points to something that it cannot do, but the mind can M is free to modify his example, but each time he does so,

L is entitled to look for defects in the revised model If M can devise a model that L cannot find fault with,

his thesis is established, otherwise it is not proven And since M necessarily cannot, his thesis is refuted X

18

Objections to Lucas1. Consistency Gödel’s theorem (and Lucas’ argument) applies only to

consistent systems If F is inconsistent, we can prove

anything, including g and ¬g(inconsistent systems are not incomplete)

Two possible responses to Lucas:1. We can’t establish our own consistency

▪ This chimes with Gödel’s second incompleteness theorem: one cannot prove the consistency of a formal system from within the system itself

▪ We might be able to prove our consistency drawing considerationsfrom outside the system, but who’s to say they are consistent?

2. We are in fact inconsistent Turing machines▪ Lucas responds that we are fallible but not systematically

inconsistent▪ Otherwise we would believe anything, but we tend to

discard inconsistencies whenever we notice them

19

Objections to Lucas2. Complexity Paul Benacerraf in God, the Devil, and Gödel (1967):

It is not easy to construct a Gödel formula In order to do so for any given formal system one must have a

solid understanding of the algorithm at work in the system The formal system the human mind might implement is likely to

be extremely complex (Hofstadter shares this view) So Lucas’ argument is a disjunction:

▪ Either no formal system encodes all human arithmetical capacity▪ Or any system which does has no axiomatic specification

which human beings can comprehend Lucas’ response:

He could be helped in producing the Gödel formulafor any given formal system

It might be difficult, but we could at leastin principle determine what the Gödelformula is for any given system

20

Objections to Lucas3. The Whiteley Sentence C.H. Whiteley (1962) claimed humans have similar

limitations to the one that Lucas’ argument attributes to machines (so perhaps we are not different from machines after all)

The Whiteley sentence:w = “Lucas cannot consistently assert this formula”

If w is true, then if Lucas asserts it he is inconsistent So one of two options:

Lucas is inconsistent (invalidating his own argument) Lucas cannot utter w on pain of inconsistency, in which case w is true

and Lucas is incomplete Counter-response:

Lucas can recognize that w is true, as there’s a point of view from which he can understand how the sentence tricks him

From this point of view Lucas can appreciate that he can’t assert the sentence, and consequently he can recognize its truth

21

Objections to Lucas4. Idealizations Lucas’ argument sets up a hypothetical scenario

involving an idealized mind and an idealized machine Some believe that once these idealizations are

rejected, Lucas’ argument falters: The output of any human mind is finite, and can thus be

programmed (Lucas: but this is not modelling in principle; actuality instead of potentiality)

Add a Gödelizing operator, generating an infinite numberof Gödel formulae (Lucas: this new system will have atransfinite Gödel formula)

Most people do not understand Gödel’s theorem, so all Lucas has shown is that a handful of competent mathematical logicians are not machines (Lucas: it only means that the argument is more obvious regarding people who do understand Gödel’s theorem)

22

Objections to Lucas5. Lucas Arithmetic David Lewis in “Lucas Against Mechanism” (1969):

“Peano arithmetic” is the arithmetic that machines can produce and “Lucas arithmetic” is the arithmetic that humans can produce

Lucas arithmetic will contain Gödel formulae while Peano arithmetic will not, so humans are not machines

But Lucas has not shown that he (or anyone else) can produce Lucas arithmetic in its entirety, which he must do if his argument is to succeed, so his argument is incomplete

Lucas’ response: All is required to disprove mechanism is to produce

a single theorem that a human can see is true but amachine cannot, which Gödel’s theorem does

23

Lucas’ Final Thought Lucas doesn't rule out the idea of intelligent computers:

“When we increase the complexity of our machines there may, perhaps, be surprises in store for us”

“Turing draws a parallel with a fission pile. Below a certain ‘critical’ size, nothing much happens: but above the critical size, the sparks begin to fly. So too, perhaps, with brains and machines…”

“Although it sounds implausible, it might turn out that above a certain level of complexity, a machine ceased to be predictable … and started doing things on its own account”

“It would begin to have a mind of its own when it was no longer entirely predictable … but was capable of doing things which we recognized as intelligent”

“But then it would cease to be a machine” “There would then be two ways of bringing new minds into the world:

▪ “The traditional way, by begetting children born of women…▪ “… and a new way by constructing very, very complicated systems of, say,

valves and relays”

24

Outline

• Brief history of AI• Lucas and Gödel• Penrose and Turing• Summary

25Sir Roger Penrose (1931-) English mathematical physicist, recreational

mathematician and philosopher Renowned for his work in mathematical

physics, in particular general relativity and cosmology

Some notable contributions: 1955: Moore-Penrose pseudoinverse 1950s: Penrose triangle 1959: Penrose stairs 1965: Penrose-Hawking

singularity theorems 1974: Penrose tiling

26Physics and Consciousness Penrose has written several books on the

connection between fundamental physics and (human) consciousness: 1989: The Emperor’s New Mind 1994: Shadows of the Mind 1997: The Large, the Small and the Human Mind

The known laws of physics are inadequate to explain the phenomenon of consciousness

Consciousness arises from quantum mechanics, and its description requires a new kind of physics

Bases his arguments on: The Emperor’s New Mind: Turing’s halting theorem

(limitations of computation) Shadows of the Mind: Gödel’s theorem

(limitations of formal systems)

27

The Emperor’s New Mind (1989)Turing’s Theorem

The Halting Problem…That’s the Question!

YesWill this

program ever halt?

No

28

Penrose’s Argument in a Nutshell1. Consider all current sound human knowledge about non-

termination2. Suppose we could reduce this knowledge to a (finite)

computer program3. Then we could create a self-referential version of this

program4. A contradiction to its correct performance can be derived

Penrose’s resolution: the second step is invalid;no program can incorporate what (finitely many)humans know

So although humans can emulate Turing machines, machines cannot simulate all humans

The human mind possesses super-Turingabilities, using undiscovered physical processes

29

Formally

Let be the set of all Turing machines mapping partially to

The domain of Mi is Let be the set of all r.e. sets

( is computable if ) Let

30

Formally Penrose’s claimed theorem: Proof:

Assume , then there is a fixed such that = Dk

So in particular(1) And by definition of ,(2) The conjunction of (1) and (2) is of the form

(¬A B)(B A), which implies A, i.e. Note that we (human mathematicians) have

convincingly demonstrated that, which means that Combining this with (1), we get

31

Where’s the Beef?

The logical error is the step“Mk(k) = was convincingly demonstrated”

What actually was demonstrated is But from assuming that holds, it cannot

be concluded that human mathematicians can find a proof for

32Turing’s Halting Problem in Brief Suppose a programming language which:

Supports programs as data Has some sort of conditional if ... then ... else ... Includes at least one non-terminating program loop

Consider the decision problem of determining whether a program X does not halt on itself:

? Suppose A is a program that claimed to return T for

(exactly) all such X Then A would fail to answer correctly regarding the

programC(Y ) := if A(Y ) then T else loop()

BecauseC(C) = T ⇔ A(C) = T ⇔ C(C) =

33

Restating Penrose Assume the following programs:

R▪ Can identify programs that diverge when fed themselves as input▪ But uses an undisclosed implementation (quantum mechanics?)

A▪ Incorporates all current, sound scientific knowledge on non-

termination (equivalently, incorporates all R’s knowledge)▪ Known implementation

G▪ “God”, an oracle that always has the correct answer

C▪ Applies Cantorian diagonalization to A to produce paradoxical behavior

K▪ Undisclosed program used by R (in its brain or logic circuitry) to

inspect programs like A

34

Restating Penrose

Let denote the set of one-input partial predicates in any standard model of computation:

For any , one can construct CA asCA(Y ) := if A(Y ) then T else loop()

is the total predicate:G(X) := [X(X) = ]

35

Restating Penrose Consider some partial predicate with the following

rule (not necessarily its only rule): return T if

program X is of the formX(Y ) := if Z(Y ) then T else loop(), where Z …andK[Z(X) ≠ T]

R is a special case of what Penrose believes humans are capable of

It answers T regarding the divergence of X(X) if it believes (via K) that the test Z(X) does not succeed

Specifically, R(CA) = T if K[A(CA) ≠ T] = T We don’t know what R will do if K returns F or does

not respond within some reasonable time frame

36

Restating Penrose The following facts are indisputable:

A(CA) = T ⇔ CA(CA) = T A(CA) ≠ T ⇔ CA (CA) = ⊥A(CA) = T ⇒ G(CA) = F A(CA) ≠ T ⇒ G(CA) = T

Then no A can always be right when asked CA

A(CA) ≠ G(CA) Since we assumed A incorporates R’s knowledge,

A(CA) = R(CA) ⇒ R(CA) ≠ G(CA) If A simulates R on CA, then R does not respond

properly So there are two options:

A(CA) ≠ R(CA) A does not simulate R on CA

A(CA) = R(CA) = ⊥ R’s knowledge is incomplete:K[A(CA) ≠ T] = ⊥

37

Options for R So R has two options:

I. R has super-Turing abilities

II. R is incomplete R(CA) = ⊥

What about other options, if R can err:III. R internalizes an untruth

K[A(CA) ≠ T ] = T but in fact ¬[A(CA) ≠ T]IV. R “acts without thinking”

K[A(CA) ≠ T ] ≠ T and in reality R(CA) = ¬G(CA) So if R gives the wrong answer on CA it’s either:

Because of the rule described earlier, and then K is unsound (III) Based on some other rule which pre-empts our rule, and then K

does not return T within some reasonable time frame (IV)

F T

F T

F T

A(CA) = R(CA)

I

IV

R(CA) = ⊥

K = T II

III

38

Getting Back to Penrose

Penrose believes (I) is the only correct option, and proposes a non-Turing-equivalent model for R

Although R is deterministic, it cannot be captured algorithmically, it is more G-like

CA(CA) = ?

39What Does This Mean for AI? If we disagree with Penrose’s option

(I), and believe that there may exist a program A that simulates R, we must choose between:II. R reasons unsoundly with KIII. R feels under pressure and answers

using a process other than KIV. R doesn’t answer at all

40(II) R Reasons Unsoundly with K Hilary Putnam:

Humans are inconsistent machines, believing falsehoods through unsound reasoning

Martin Davis: “No human mathematician can claim

infallibility. We all make mistakes!” “There is nothing in Gödel’s theorem to

preclude the mathematical powers of a human mind being equivalent to an algorithmic process that produces false as well as true statements”

41

(III) R Answers without K John McCarthy:

“Much of Penrose’s reasoning is nonmonotonic, e.g. preferring the simplest explanation”

“But his methodology doesn’t allow for nonmonotonic reasoning by the program”

There is an A = R which answers incorrectly, but Penrose looks at A′ which acts like R for which only monotonic K is consulted, and A’ ≠ R

J.R. Lucas: “Our inconsistencies are mistakes rather than set

policies” “They correspond to the occasional malfunctioning of a

machine, not its normal scheme of operations”

42

(IV) R Doesn’t Answer R could be “inadequate”, not giving an answer about CA, and

maybe that’s alright, cause we are also “inadequate” David Chalmers:

“Perhaps we are sound, but we cannot know unassailably that we are sound”

Hilary Putnam: “There is an obvious lacuna: the possibility of a program … which is not

simple enough to appreciate in a perfectly conscious way is overlooked” Geoffrey LaForte, Patrick J. Hayes, and Kenneth M. Ford:

“One can show … that Penrose’s notion of what it is to know oneself to be sound cannot itself be sound”

“Humans may be unable to know that they are consistent” Arnon Avron:

“We cannot fully analyze a complicated learning machine, let alone the human mind”

“One cannot establish one’s own self-consistency”

43

Strange Loop An interesting case: suppose A is designed

to parrot R when asked whether CA(CA) = Note that when anyone asks about CA, CA

consults A, which turns the question over to R So what does R answer?

R says “yes” to A A says “yes” to CA CA(CA) = T R says “no” to A A says “no” to CA CA(CA) =

No matter what R says, it will be wrong The only sound alternative for R is to

“take the Fifth”: R(CA) =

44

Shadows of the Mind (1994)Gödel’s Theorem

45

Penrose’s Gödelian Error Suppose:

A human H has a computational mind associated with the program P

H is made aware of P which we are using to “compute his brain” We may use a formal mathematical system F to logically deduce

the behavior of P, and hence H’s behavior, beliefs and thoughts So F encapsulates all of H’s knowledge and beliefs

Penrose argues: If H believes statement X, then since F encapsulates all of H’s

beliefs, it must be possible to prove X in the system F H will surely believe that the system F is sound So H will believe (by Gödel's theorem) that g is true But since g is not a theorem of F, it follows that F cannot after all

encapsulate all of H’s beliefs So H cannot have a computational mind

46

Penrose’s Gödelian Error Penrose’s confusion:

If H believes X (on day 1) then it is necessary that F deduces thatH believes X on day 1, but it is not necessary that F deduces X!

If on day 2 H changes his mind and believes the opposite ¬X, then it is necessary that F deduces that H believes ¬X on day 2 (and not ¬X)

If on day 3 H goes crazy then it is necessary that F deduces that H is crazy on day 3; it is not necessary that F deduces all sorts of crazy formulae

Although F cannot deduce g, there is no reason why F cannot deduce that H believes g

Gödel's theorem doesn’t say anything about what can be proved concerning the state of H’s mind!

The essential point: H’s beliefs do not form part of the deductive system F If H believes two contradictory statements X and Y, these statements

are not theorems of F and so it does not follow that F is inconsistent, as it would be if X and Y were theorems

47

Feferman’s Criticism Solomon Feferman has a common cause with Penrose:

Opposes the dominant computational model of the mind Human thought, and in particular mathematical thought, is

not achieved by mechanical application of algorithms Rather, by trial-and-error, insight and inspiration, in a

process that machines will never share with humans But has a lot of criticism:

Penrose extends his argument too far in areas such as mathematical soundness and consistency, providing ammunition for the computational-mind camp

Feferman finds several technical and logical errorsin Penrose’s reasoning

But then concludes that Penrose’s case wouldnot be altered by correcting these flaws

48

Feferman’s Criticism The computational-mind argument stresses the equivalence

between Turing machines and formal systems But the model of mathematical thought using formal systems

is closer to the nature of human (and particularly mathematical) thought

The Turing machine model: given a problem, human reason would plug away, applying the same algorithm indefinitely, in the hope of finding an answer

But it’s ridiculous to think that mathematics is done this way The basis of mathematical success is trial-and-error

reasoning, insight and inspiration, based on prior experience – but not on general rules

A mechanical approach is only appropriate after an initial proof has been arrived at, for checking and validating it

49

Outline

• Brief history of AI• Lucas and Gödel• Penrose and Turing• Summary

50

So What Does Gödel Have to Say? “So the following disjunctive conclusion is

inevitable: “Either mathematics is incompletable in this sense,

that its evident axioms can never be comprised in a finite rule, that is to say, the human mind (even within the realm of pure mathematics) infinitely surpasses the powers of any finite machine…”

“… or else there exist absolutely unsolvable diophantine problems of the type specified…”

Lucas: “It is clear that Gödel thought the second disjunct false”

51

And What About Turing? “There are certain things that [any digital computer]

cannot do” “If it is rigged up to give answers to questions as in

the imitation game, there will be some questions to which it will either give a wrong answer, or fail to give an answer at all however much time is allowed for a reply”

Ordinal logics as a way to “escape” Gödel: In pre-Gödel times it was thought by some that it would be

possible to carry this to such an extent that the necessity for intuition would be eliminated

Not all the steps in a proof are mechanical, some are intuitive

“If a machine is intelligent, it must also be fallible”

52

And What About Turing? “Although it is established that there are limitations to the powers of

any particular machine, it has only been stated, without any sort of proof, that no such limitations apply to the human intellect”

“Whenever one of these machines is asked the appropriate critical question, and gives a definite answer, we know that this answer must be wrong, and this gives us a certain feeling of superiority. Is this feeling illusory?”

“It is no doubt quite genuine, but I do not think too much importance should be attached to it”

“We too often give wrong answers to questions ourselves to be justified in being very pleased at such evidence of the fallibility on the part of the machines”

“Our superiority can only be felt on such an occasion in relation to the one machine over which we have scored our petty triumph”

“There would be no question of triumphing simultaneously over all machines”

“There might be men cleverer than any given machine, but then again there might be other machines cleverer again, and so on”

53

Final Thought AI began as a godly feat of endowing the inanimate with life Formal reasoning put the emphasis on mechanizing human

thought instead of body Rational thinking strived to eliminate the need for a “soul”

in describing the mind When AI was born, the mechanists adopted either

reductionism or holism, but in any case there was no “soul” But just at the same time, mathematics and philosophy

showed that there is still a gap between mind and machine What is this gap? The human soul? Undiscovered physics?

Can we ever actually know? So we now know the theoretical limits of AI… But it seems that how it will evolve is a question of what

you believe in

Thank You!