Iftach Haitner and Yishay...

Computational Models - Lecture 7 1

Iftach Haitner and Yishay Mansour.

Tel Aviv University.

May 7/9, 2012

1Based on slides by Benny Chor, Tel Aviv University, modifying slides by MauriceHerlihy, Brown University.

Iftach Haitner and Yishay Mansour (TAU) Computational Models Lecture 7 May 7/9, 2012 1 / 56

Talk Outline

Church-Turing Thesis

Enumerators

Decidability vs. Enumerability

Encoding of Turing Machines and Universal Turing Machines

The Halting/Acceptance problem

Beyond Enumerable and co-Enumerable

Sipser’s book, 3.2, 3.3, 4.1, 4.2


Part I



What is an Algorithm?

Informally:1 A recipe

2 A procedure3 A computer program4 Who cares? I know it when I see it :-(5 The notion of algorithm has long history in Mathematics (starting

with Euclid’s gcd algorithm), but not precisely defined until 20’thcentury

Informal notions rarely questioned, still they were insufficient


Computation Model

Many models have been proposed for general-purposecomputation. Remarkably, all “reasonable” models were shown tobe equivalent to Turing machines.

The notion of an algorithm is model-independent!

We don’t really care about Turing machines per se.

We do care about understanding computation, and because oftheir simplicity, Turing machines are a good model to use.


Models Equivalent to TM

All “reasonable” programming languages (e.g., Java, Pascal, C,Python, Scheme, Mathematica, Maple, Cobol,. . . ).

λ-calculus of Alonzo Church

Turing machines of Alan Turing

Recursive functions of Godel and Kleene

Counter machines of Minsky

Normal algorithms of Markov

Unrestricted grammars

Two stack automata

Random access machines (RAMs)...



“The intuitive notion of reasonable models of computationequals Turing machine algorithms”.


“Wild" Models of Computation

Consider MUntel’s ℵ-AXP10 c© processor (to be released ...)

Definition 1 ( ℵ-AXP10 c©)

Like a Turing machine, except

Takes first step in 1 second.

Takes second step in 1/2 second.

Takes i-th step in 2−i seconds . . .

After 2 seconds, the ℵ-AXP c© decides any enumerable language!

Question 2

Does the ℵ-AXP c© invalidate the Church-Turing Thesis?


Part II

Enumerators


Enumerators

A language is enumerable, if it is accepted by some Turing Machine.

Question 3But why enumerable?

1 010

aaabba

Definition 4 (enumerator)

A TM M is an enumerator for a language L, if M “outputs" all thestrings in L and nothing else.


Having Enumerator ⇔ Being in RE

Intuition 5Enumerator is a program with no input, writes to a write-only file, andeach line is interpreted as a word that it outputs.

Theorem 6A language is in RE iff it has an enumerator.

Will show

If E enumerates language L, then some TM M accepts L.

If M accepts L, then some enumerator E enumerates it.


Having Enumerator =⇒ Being in RE

Claim 7If a TM E enumerates a language L, then some TM M accepts L.

Proof:

Algorithm 8 (TM M)

On input w , run E .

Every time E outputs a string v :

If v = w , accept.

♣


Being in RE =⇒ Having Enumerator

Claim 9If a TM M accepts L, then some enumerator TM E enumerates L.

Proof: Let s1, s2, s3, . . . be a list of all strings in Σ∗ (e.g., strings inlexicographic order).

Algorithm 10 (TM E)

Repeat the following for i = 1, 2, 3, . . .Run M for i steps on each input s1, s2, . . . , si .

For any accepting computation, output the corresponding s.

♣

Note that with this procedure, each output is duplicated infinitely often.

Question 11Can this duplication be avoided?


Having “In order" Enumerator ⇔ Being in R

Theorem 12A language L is decidable if and only if there is some enumerator Ethat enumerates L in lexicographic order.

Proof: ? Left as an exercise. ♣


Part III




RE – the class of enumerable languages

coRE – the class of languages whose complement is enumerable

R – the class of decidable languages.

L ∈ R =⇒ L ∈ REL ∈ R =⇒ L ∈ RThm (next slide): R = RE ∩ coRE


Theorem R = RE ∩ coRETheorem 13R = RE ∩ coRE .

Proof:R ⊆ RE ∩ coRE

I L ∈ R =⇒ L ∈ REI L ∈ R =⇒ L ∈ R =⇒ L ∈ RE =⇒ L ∈ coRE

R ⊇ RE ∩ coREFor L ∈ RE ∩ coRE , let M1 be a TM that accepts L, and let M2 be aTM that accepts L.

Algorithm 14 ( M - a decider for L)

Input: w .

I Run both M1 and M2 in “in parallel".

I Accept if M1 accepts

I Reject if M2 accepts

We next show that M decides L.♣Iftach Haitner and Yishay Mansour (TAU) Computational Models Lecture 7 May 7/9, 2012 17 / 56

M decides L

Claim 15M decides L

Proof:

Every string is in L or in L (of course not in both).

Thus either M1 or M2 accepts the input w .

Since M halts whenever M1 or M2 accepts, M always halts (andhence is a decider).

Moreover, M accepts strings in L and rejects strings in L.

♣


Emulating TM’s In Parallel

Question 16What does it mean to emulate M1,M2 in parallel?

Answer: M has two tapes, one for each machine.

Algorithm 17 (TM M)

Do (forever)

1 Emulate the next step of M1

2 Emulate the next step of M2

3 If this is accepting configuration for some Mi , halt and return i .


Revised View of the World of Languages

context free

regular

decidable

enumerable


Part IV

Encodings and Universal TM


Encodings

Input to a Turing machine is a string of symbols.

We want algorithms that work on graphs, matrices, polynomials,Turing machines, etc.

Need to choose an encoding for objects (can often be done inmany reasonable ways).

Sometimes it is helpful to distinguish between X , the object, and〈X 〉, its encoding.


Encoding of Turing Machines

Just like any piece of code in any programming language, Turingmachines can be encoded as strings.Such encoding will enable us

To check (by an algorithm) that a given string is a legal encodingof a TM.

To build a universal machine that can read such encoding andemulates the encoded TM on any input string.


Standard Encoding of Turing Machines

Definition 18 (Encoding of 〈M〉 of a TM M)

Let M = (Q,Σ, Γ, δ, q0, qa, qr ) be a TM. Assume wlg. that

Q = {q1, . . . qm}, where q0, qa and qr are indicated by states q1,q2 and q3 respectively.

Γ = {γ1, . . . , γs}, where 0, 1 and are indicated by γ1, γ2 and γ3.

The directions L and R be indicated by D1 and D2.

To encode M, we only encode the transition function δ. For each ruleδ(qi , γj) = (qk , γ`,Db), we add the string 0i10j10k10`10b.

Different rules are separated by 11.

Fact 19There exists a TM (called universal TM) that on input 〈M,w〉 (encodedby (1100∗100∗100∗100∗100∗)∗111(0 ∪ 1)∗, can check that 〈M〉encoded a TM, and can emulates M(w).


The Universal Turing Machine


Universal Turing Machines

Algorithm 20 (Universal TM U)

On input (〈M,w〉, where 〈M,w〉 are binary strings separated by 111.

Checks that 〈M,w〉 is a proper encoding of a TM.Emulate M(w)(how?)

I Accept, if M enters its accept stateI Reject, if M enters its reject state

Notice that as a consequence, if M on input w enters an infinite loop,so does U on input 〈M,w〉.


Universal Turing Machines (2)

The universal machine U obviously has a fixed number of states(100 should do).

Despite this, it can simulate machines M with many more states.

Universal machines inspired the development of stored-programcomputers in the 40s and 50s.

Most of you have seen a universal machine, and have even usedone!


Universal Turing Machines (3)

For example, Dr. Scheme (interpreter) is a universal Schememachine.

It accepts a two part input: “Above the line” – the program(corresponding to 〈M〉), and “below the line” the input to run it on(corresponding to w).


Part V

The Acceptance & Halting Problems


The Acceptance & Halting Problems

Of the most philosophically important theorems of the theory ofcomputation.

Computers (and computation) are not omnipotent – they arelimited in a very fundamental way.Many common problems are unsolvable, e.g., does a program sortan array of integers?

Note that these problems are well defined: both program andspecification are precise mathematical objects.

Hey, proving program ∼= specification should be just like provingthat triangle 1 ∼= triangle 2 . . .

Well, this is not the case!


CFG, NFA, DFA RemindersLet ECFG = {〈G〉 | G is a CFG and L(G) = ∅}

We saw that ECFG is a decidable language.

ACFG = {〈M,w〉 | M is a PDA accepting the string w}.Saw that the language ACFG is decidable.

ANFA = {〈M,w〉 | M is an NFA accepting the string w}.

ADFA = {〈M,w〉 | M is a DFA accepting the string w}.

Saw both ANFA and ADFA are also decidable.

What would happen with Turing Machines?

ATM = {〈M,w〉|M is a TM that accepts w}

Theorem 21 (The Acceptance Problem is undecidable)

ATM is undecidable.


The Acceptance Problem

ATM = {〈M,w〉|M is a TM that accepts w}

Before approaching the proof of undecidability, we first prove

Theorem 22ATM is recursively enumerable (namely in RE).

Proof: The universal machine accepts ATM. ♣


Proving Thm 20

We prove Thm 20 by contradiction.Suppose a TM, H, is a decider forATM. Namely,

H(〈M,w)〉) ={

accepts if M accepts wreject if M does not accept w

We construct a new TM, D, with H as a subroutine.

Algorithm 23 ( D)

On input 〈M〉Run H on input 〈M,M〉.Output the opposite of what H outputs:

I Reject if H accepts, andI Accept if H rejects.


Proving Thm 20 cont.

What happens if we run D on its own description?

D(〈D〉) ={

reject if D accepts 〈D〉accept if D rejects 〈D〉

Oh, oh...Or, more accurately, a contradiction (to what?) ♣


Explaining the contradictionConsider the algorithm D on input < D >.

Algorithm 24 ( D)

On input 〈D〉Run H on input 〈D,D〉.Output the opposite of what H outputs:

I Reject if H accepts (i.e., D accepts < D >), andI Accept if H rejects (i.e., D does not accepts < D >).

Question 25What should be the output of D(< D >)?

Answer:If D accepts < D > then H(< D,D >) accepts, so D(< D >)rejects.If D does not accepts < D > then H(< D,D >) rejects, soD(< D >) accepts.contradiction


Self Reference

Don’t be confused by the notion of running a machine on its owndescription!

Actually, you should get used to it.

Notion of self-reference comes up again and again in diverseareas.

This notion of self-reference is the basic idea behind Gödel’srevolutionary result.

Compilers do this all the time . . . .


A Non-Enumerable Language

We already saw a non-decidable language: ATM.

Can we do better (i.e., worse)?

We now display a language that isn’t even recursively enumerable. . . .

Corollary 26

If L is not decidable, then either L or L is not enumerable.

Proof: Assume otherwise, then by Thm 12 L is decidable. ♣

Corollary 27

ATM is not Enumerable


The Halting Problem

HTM = {〈M,w〉|M is a TM and M halts on input w}

Theorem 28HTM is undecidable.

Proof idea:

Similar to ATM.

Alternatively, by a reduction to ATM.


The World as we (currently) Know It

enumerableco-enumerable decidable

A TMA DFAA TM

???

Question 29Are there any languages in the area marked ??? ?

Answer: Yes, ThinkL = {< M1 >,w1, < M2 >,w2|w1 ∈ L(M1),w2 6∈ L(M2)}.


Part VI

Beyond Enumerable and co-Enumerable


Comparing Sizes of Sets

Suppose A and B are two sets, and we wish to compare theirsizes.

If both A and B are finite, we can count how many elements eachof them has, and compare the numbers.

This method does not generalize to infinite sets.

Alternatively, we can pair the elements of A and B. If they pairperfectly, they have equal sizes.


Correspondence

Question 30What does it mean to say that two infinite sets are of the same size?

Answered by Georg Cantor in 1873: Pair them off.

Definition 31A map f : A → B is a correspondence if f satisfies:

One-to-one: if a1 6= a2 then f (a1) 6= f (a2).

Onto: for every b ∈ B, there is an a ∈ A such that f (a) = b.

Definition 32A and B are of the same size, if there is a correspondence from A to B.


Correspondence – Example

Claim 33The set N of natural numbers has the same size as the set E of evennumbers.

Proof: The mapping f (i) = 2i is a correspondence from N to E. ♣

Remark 34The set E is a proper subset of the set N, yet they are the same size!


Countable Sets

Definition 35A set A is countable if

Either A is finite, or

A has the same size as N.

We have just seen that E is countable.

An infinite countable set is sometimes said to have size ℵ0.

Claim 36Assuming ∃ a one-to-one mapping from a set S to a countable set,then S is countable.

Proof: Exercise. . .♣


Integers are Countable

Claim 37The set Z of integers is countable.

Proof: Define f : N → Z by

f (i) ={

i/2 if i is even−(bi/2c+ 1) if i is odd

♣


Rational Numbers are Countable

Theorem 38

Q = {mn | m, n ∈ N} is countable.

Proof’s Idea:

List Q as 2-dim array

Begin counting with the first row . . .

Why doesn’t this work?


Rational Numbers are Countable 2

1/2 1/3 1/4 1/51/1

2/1

3/1

4/1

5/1

2/2 2/3 2/4 2/5

1/2 3/3 3/4 3/5

4/2 4/3 4/4

5/35/2

Enumerate numbers along northeast and diagonals, skippingduplicates.

Does this mean that every infinite set is countable?


The Real Numbers are Uncountable

Every real number has a decimal representation.

For example, π = 3.1415926 . . .,√

2 = 1.4142136 . . ., and0 = 0.0000000 . . ..

Theorem 39R– the set of all real numbers – is uncountable.

R is sometimes said to have size ℵ1.

This is Cantor’s historic proof, which introduced the diagonalizationmethod.


The Real Numbers are Uncountable 2

Assume exists a correspondencebetween N and R:

n f (n)1 3.14159. . .2 55.55555. . .3 40.18643. . .4 15.20601. . .

We show that ∃ number x not inthe rhs of the list.

First fractional digit of f (1) is1, so pick first fractional digitof x to be something else(say, 2).

Second fractional digit off (2) is 5, so pick secondfractional digit of x to besomething else (say, 6).

and so on . . .

x = 0.2671 . . .

This proof’s technique is called Diagonalization


Back to Languages & TMs

A similar proof shows there are languages that are notenumerable.

Actually (we show later), the set of all languages is uncountable!

Since the set of Turing machines is countable (we show next)“most” languages are not enumerable (and not co-enumerable)


The Set of all Turing machines Is countable

Claim 40For every finite Σ, the set Σ∗ is countable.

Proof: List strings of length 0, then length 1, then 2, and so on. Thisexhausts all of Σ∗.

The union of countably many finite sets is countable. ♣

Claim 41The set of all Turing Machines is countable.

Proof: Each TM M has an encoding as a string 〈M〉. Therefore there isa one-to-one mapping from the set of all TMs into (but not onto) Σ∗.

Since Σ∗ is countable, so is the set of all TMs. ♣


The Set of All Languages Is Uncountable

Claim 42Let B be the set of infinite binary sequences. Then B is uncountable.

Proof: Diagonalization argument, essentially identical to the proof thatR is uncountable. ♣

Claim 43Let L be the set of all languages over alphabet Σ. There exists acorrespondence from L to B

Hence L is uncountable.


The Correspondence χ : L → B

Definition 44 (The Characteristic function χ)

Let Σ∗ = {s1, s2, s3, . . .} (in lexicographic order).The i-th bit of χ(L) is 1 iff si ∈ L.

Example 45

Σ∗ = {ε, 0, 1, 00, 01, 10, 11, 000 . . .}L = { 0, 00, 01, 000 . . .}χ(L) = {0, 1, 0, 1, 1, 0, 0, 1 . . .}

Claim 46χ is one-to-one and onto (why?).

Hence, χ is a correspondence, yielding that L is uncountable.


TMs vs. Languages

The set of all Turing machines is countable.

The set of all languages is uncountable.

Therefore, there are languages outside RE (why?).

Moreover, there are languages outside RE ∪ coRE (why?).

This is an existential proof – it does not explicitly show any suchlanguage.


Reflection on The proof that ATM /∈ R

The proof that the acceptance problem is undecidable is actuallydiagonalization in transparent disguise.

〈M1〉〈M2〉〈M3〉〈M4〉 . . .M1 accept acceptM2 accept accept accept acceptM3

M4 accept accept...

Entry (i , j) is accept if Mi accepts 〈Mj〉, and

blank if Mi rejects or loops on 〈Mj〉.


Reflection on The proof that ATM /∈ R, 2

〈M1〉〈M2〉〈M3〉〈M4〉 . . .M1 accept reject accept rejectM2 accept accept accept acceptM3 reject reject reject rejectM4 accept accept reject reject...

......

...

In this table, entry (i , j) states whether H accepts 〈Mi , 〈Mj〉〉.

D(〈M〉) computes the opposite of H(〈M,M〉)


Reflection on The proof that ATM /∈ R, 3

Now we add D to the table.

D(〈M〉) computes the opposite of H(〈M,M〉)〈M1〉〈M2〉〈M3〉 . . . 〈D〉

M1 accept reject acceptM2 accept accept acceptM3 reject reject rejectM4 accept accept reject...

.... . .

D reject reject accept ???...

... . . .

H is a TM =⇒ D is TM =⇒ occurs on the list M1,M2, . . ..

Hence, D(〈D〉) 6=D(〈D〉) !


Iftach Haitner and Yishay...

Documents

Transcript of Iftach Haitner and Yishay...