Computability and complexity over discrete

structures(Notes for an FMB course)

(SECOND DRAFT)

Viggo Stoltenberg-Hansen∗

Department of Mathematics, Uppsala University, Box 480,S-751 06 Uppsala, Sweden

March 6, 2007

Contents

1 Introduction 1

2 Model of computation 3

3 Basic theory of computability 93.1 Universal Turing machines . . . . . . . . . . . . . . . . . . . . 93.2 The halting problem . . . . . . . . . . . . . . . . . . . . . . . 123.3 Semicomputable sets and completeness . . . . . . . . . . . . . 143.4 Nondeterministic Turing machines . . . . . . . . . . . . . . . 18

4 Complexity 194.1 Basic definitions and results . . . . . . . . . . . . . . . . . . . 204.2 Complexity hierarchies . . . . . . . . . . . . . . . . . . . . . . 23

5 Feasible computations 26

6 The Cook-Levin theorem and related results 31

7 Further reading 36

References 37∗email: viggo@math.uu.se

1

1 Introduction

Computability theory is the study of which sets and functions are algorithmi-cally computable and, further, algorithmical relations between sets and func-tions which themselves need not be computable. Complexity theory studiessimilar questions except that the notion of algorithmically computable isreplaced by feasably computable or computable in practice.

The first necessary step in such a study is to make the intuitive notionsof “algorithmically computable” and “feasably computable” mathematicallyprecise. This has been a non-trivial problem but now there is a generalagreement at least for the first concept.

The intuitive notion of algorithm has been around troughout the historyof mathematics. Here is a vague attempt to capture its meaning. An algo-rithm is a mechanical procedure given by a finite set of instructions. Theprocedure in the associated computation consists of discrete steps. What isdone at each step may depend and only depend on the input given to theprocedure, what has been done on the previous steps, and on the finite setof instructions.

The above is, of course, an incomplete description and hence not satis-factory. Nonetheless we have a rather clear notion of what an algorithm is.We recognise an algorithm when we see one! However, a problem arises ifwe want to show that there is no algorithm computing a certain function.Then we have to quantify over the set of all algorithms and show that noneworks. For this we need a precise mathematical definition of that set.

In 1936 the notion of an algorithmically computable function was cap-tured in a mathematical definition by A. M. Turing [12] and others. It gaverise to the following thesis.

Church-Turing Thesis: A function is algorithmically computable if, andonly if, it is computable by a Turing Machine.

Turing machines will be introduced in Section 2. There are many othermathematical definitions of algorithmically computable functions but all ofthese have been shown to be equivalent to that of Turing computable, (i.e.computable by a Turing machine). This fact is strong support, according tosome, of the Church-Turing thesis. Further support is that no algorithmi-cally computable function (remember that we recognise one when we see analgorithm for it) has been found which is not computable by a Turing ma-chine. Finally, the class of Turing computable functions has a strong closureproperty: there exists a universal Turing machine, i.e. one machine whichcan compute whatever another machine can compute (see Section 3.1).

The question what the feasible computations should be is more delicatein the sense that there is no clear cut answer analogous to the Church-Turing thesis. In 1964 A. Cobham [3] essentially proposed that the feasablycomputable functions (sets) are those that can be computed (decided) by a

2

Turing machine in polynomially many steps. To be slightly more precise,P is the class of sets or problems (subsets of some formal language) withthe following property: A set A is in P if there is a Turing machine M anda polynomial p(x) such that for each word w in the formal language, Mterminates in less than p(|w|) steps, and w ∈ A if and only if the output ofM is “yes”. (|w| denotes the number of symbols in the word w).

The class P is now normally considered as the set of problems whichare decidable by feasible computations. Similarly, the class of functionswhich are feasably computable are those that can be computed by a Turingmachine in polynomially many steps (often called polynomial time). We say“normally” since it is clearly the case that the intuitive notion of feasablycomputable functions cannot be made exact. For example, a set in P mayneed a polynomial of high degree. Is it then computable in practice?

A pragmatic answer is that “natural” problems shown to be in P tendto be associated with polynomials of low degree. An exception is the recentcelebrated result of Agrawal, Kayal and Saxena [1] showing that the set ofprimes is in P. The polynomial produced in [1] is (roughly) of degree 12.

There are many natural problems which are decidable in polynomial timeby non-deterministic Turing machines. A nondeterministic Turing machineis one which at each step has several (finitely many) choices of action. Aword w is accepted by a nondeterministic Turing machine if there is atleast one computation path which accepts w. The class of sets acceptedby nondeterministic Turing machines coincides with the class accepted bydeterministic Turing machines. The main question to be solved in this area,though very likely not in this course, is whether the two classes coincidewhen restricted to polynomial time. This is the P = NP question.

2 Model of computation

As a model of computation we choose Turing Machines (TM) since thecomplexity notions introduced in Section 4 become very natural. Othermodels of computation often considered, such as the RAM machine, areequivalent to the Turing machine model at least up to polynomial time. Inthis sense the theory of complexity is robust, i.e. the major results are notdependent on the particular model of computation.

A Turing machine operates on a set of words over a finite alphabet. Analphabet is any non-empty finite set. We usually denote alphabets by uppercase Greek letters such as Σ and ∆. Think of the elements in an alphabetas symbols or letters. A word over an alphabet Σ is a finite sequence ofsymbols from Σ. The empty sequence, i.e. the word consisting of no symbols,is denoted by ε. The length of a word w, i.e. the number of symbols in w,is denoted by |w|. The set of all words over Σ is denoted by Σ

∗. Often we

let a, b, c, . . . vary over Σ while x, y, z, u, v, w, . . . vary over Σ∗.

3

There is a natural binary operation on Σ∗. Given x, y ∈ Σ

∗define

x · y = xy, the concatenation of x and y. Here the right hand side is thesequence consisting of x followed by y. For example, if x = ab and y = cthen xy = abc. Note that concatenation is associative. Furthermore we haveε · x = x · ε = x, that is ε is the identity with respect to concatenation. Itfollows that the structure (Σ

∗, ·, ε) is a monoid. In fact, (Σ

∗, ·, ε) is the free

monoid over n generators, where Σ has n elements.A (formal) language over Σ is a subset of Σ

∗. Suppose A is a language

over Σ and B is a language over ∆. Then we define A∪B, A∩B, and A−Bin the usual set theoretic way. We can also define the concatenation of Aand B by

A ·B = {xy : x ∈ A and y ∈ B}.

All of these are languages over Σ ∪∆.A Turing machine is a primitive version of an idealised computer. It is a

finite machine in the sense that the central processor contains a finite numberof instructions manipulating a (remember finite) alphabet Σ. However, it isidealised in the sense that there is no a priori upper bound on the amountof information that can be stored on its “hard disc”. The storage device isa tape divided into squares. The tape contains infinitely many squares eachcontaining a symbol from the alphabet Σ] = Σ ∪ {]}, where ] is a specialsymbol not in Σ denoting a blank square. At each step of a computation atmost finitely many squares will contain symbols from Σ.

a b c b c a a c b c

M

Figure 1

A Turing machine can read symbols from the tape, write symbols on thetape and move (its reading head) along the tape. The machine works indiscrete steps. At each step the machine is in one of finitely many states.(This is the sense in which the machine is finite.) At a particular step themachine reads a symbol. Then, depending on what symbol it has read andwhat state it is in, the machine

• enters a new state,

• writes a new symbol on the square just read, and

4

• moves the reading head one square to the right or one square to theleft.

The action of a machine at a step is determined by its transition function.It takes a state and a symbol as arguments and gives out a state, a symboland a direction as values. The intuition of the transition function is the onedescribed above.

There is one more point to consider, namely, when does the machine stop.For this we divide the states of a machine into active states and passive orfinal states. When the machine enters a passive state it simply stops.

Here is the mathematical definition.

Definition 2.1. A Turing machine M is a five-tuple

M = (Q,Σ, δ, q0, F )

where

(i) Q is a finite set (the set of states);

(ii) Σ is an alphabet;

(iii) q0 ∈ Q (the initial state);

(iv) F ⊆ Q (the set of final states);

(v) δ: (Q− F )× Σ] → Q× Σ] × {R,L} (the transition function).

The intuitive interpretation of R and L is that the reading head movesone step to the right or to the left, respectively.

At each step of a computation we can describe the status of the machineby a finite word. Such a description is called a configuration or instantaneousdescription (ID). Let M = (Q,Σ, δ, q0, F ) be a Turing machine. An ID is atriple (w, q, v), where w, v ∈ Σ

∗] and q ∈ Q. The intuitive interpretation of

(w, q, v) is that the tape contains the word wv, all squares to the right orleft of wv contain only the symbol ] for blank, the machine is in state q, andthe reading head is over the leftmost symbol or letter in v. (If v = ε thenthe reading head is over the first square to the right of w.)

Now we define a relation ` between configurations describing one stepin a computation. (Remember our convention on variables.)

Definition 2.2. Given a Turing machine M = (Q,Σ, δ, q0, F ) we define `between configurations for M by

(wa, q, bv) `{

(wac, p, v) if δ(q, b) = (p, c, R)(w, p, acv) if δ(q, b) = (p, c, L).

5

We extend the relation ` to its transitive closure∗`. Thus we define

n` in-

ductively by (w, q, v)0` (w, q, v) and (w, q, v)

n+1` (x, p, y) if there is (x′, p′, y′)

such that(w, q, v)

n` (x′, p′, y′) and (x′, p′, y′) ` (x, p, y).

Now we define

(w, q, v)∗`(x, p, y) if ∃n((w, q, v)

n` (x, p, y)).

Intuitively, (w, q, v)n` (x, p, y) means that the Turing machine M al-

ters the configuration (w, q, v) to (x, p, y) in exactly n steps. Furthermore,

(w, q, v)∗`(x, p, y) if M alters the configuration (w, q, v) to (x, p, y) in a finite

number of steps. Note that these relations are dependent on the particularTuring machine M . Thus a more appropriate notation would be `M for `but we omit the subscript M for ease of readability.

We can now give a precise definition of what it means for a partialfunction f : Σ

∗ → Σ∗

to be Turing machine computable. Intuitively f isTuring machine computable if and only if there is a Turing machine M suchthat for each w ∈ Σ

∗, if M starts with w written on the tape, then M stops

if and only if f(w) is defined and in that case f(w) is written on the tape.We use the notation f(w)↓ for f(w) being defined. Similarly, f(w)↑ meansthat f(w) is undefined. Here is the precise definition.

Definition 2.3. Let Σ be an alphabet. A partial function f : Σ∗ → Σ

∗is

Turing machine computable (or just computable) if there is a Turing machineM = (Q,∆, q0, δ, F ) such that Σ ⊆ ∆ and for each w ∈ Σ

∗,

(i) f(w)↓ ⇐⇒ (∃p ∈ F )((ε, q0, w)∗`(x, p, y)), and

(ii) f(w)↓ =⇒ (ε, q0, w)∗`(ε, p, f(w)) for some p ∈ F .

(i) says that M stops if and only if f(w)↓ and (ii) says that the valuecan be read off from the tape starting with the symbol under the readinghead. The starting position of M is that the argument shall be on the tapewith the reading head over the left most symbol.

Note that the alphabet for M is ∆ which can exceed Σ. It is often conve-nient for a machine to use extra symbols in order to simplify a computation.

In the above definition we have made a definite choice of conventions.It should be clear that we could have made other equivalent choices. Forexample, we could require that we start the machine on the right mostsymbol. Or we could have restricted F to contain only one element (usuallydenoted h for halt). Other variants are to have the tape infinite in onlyone direction or to have finitely many tapes and a reading head for eachtape. It is an easy but boring exercise to show that these variants are all

6

equivalent. For this reason we will often be sloppy and deviate from ourchosen conventions.

Before giving some examples we define a very useful subclass of Turingmachines. A Turing machine M = (Q,Σ, δ, q0, F ) is normal if each state isassociated with a fixed direction. This means that whenever the machineenters a fixed state it moves in the associated direction. Formally, for eachp ∈ Q,

δ(q1, a1) = (p, b1, D1) and δ(q2, a2) = (p, b2, D2) =⇒ D1 = D2.

Example 2.4. We construct a Turing machine which computes the functionf : N → N defined by f(n) = 2n. First we need to decide on a representationof the natural numbers. We choose a unary representation over the alphabetΣ = {1}. Thus the number n is represented by the word 11 · · · 1 consistingof n ones. Let the Turing machine M have alphabet {1, A}. The algorithmis simply to write a new 1 for each 1 in the input. The symbol A is used toindicate that we have taken care of a 1. Reading a 1 we write A and searchleft to first blank ]. Write A and then search right for first 1 and then startover again. When a blank appears we go left changing back every A to 1and then we stop.

This algorithm is more conveniently described by the state diagram inFigure 2.

L LR1A

#

#

h

A1

A #

Figure 2

To make diagrams of Turing machines meaningful and more readable wefollow the conventions described in Figure 3.

7

Conventions

initial state

a a

such arrows are deleted

q Rp

ba M is in state q, if M reads a then M

writes b, enters state p and moves right

aa written as a

Figure 3

Example 2.5. This example shows how a Turing machine can locate anentry in a table. Let Σ = {A,B, c, d, x, y}. (Note that here x, y are symbols!)Let the input tape be

· · · ]wxcn0xcn1x · · ·xcnmx] · · ·

where w ∈ {A,B}∗ is a binary representation for a number ≤ m. (A standsfor 0 and B for 1, so (ABB)2 = 3.)

Suppose (w)2 = i. Then the output tape should be:

· · · ]vydn0ydn1 · · · ydni−1ycnixcni+1x · · ·xcnmx] · · ·

Note that the ith entry starts with the first c. Also note that the originaltable can easily be restored by another machine. The machine in Figure 4performs the desired computation.

R L

y

A

d

c x

B #

h

B

A

Figure 4

8

Example 2.6. This machine converts a unary representation of a numberto a 5-ary representation. We assume 0, 1, 2, 3, 4 is used in the 5-ary systemwhile c is used in the unary system. The input tape is:

· · · ]ycnx] · · ·

The output tape should be:

· · · ]wydnx] · · ·

where w ∈ {0, 1, 2, 3, 4}∗ and (w)5 = n.A machine performing this feat is given in Figure 5.

L R0

4

d

1

2

3

4

xh

01

23

c

#

1

Figure 5

As an exercise change Example 2.6 to deal with binary representationinstead of 5-ary representation. Then combine Example 2.5 and Example2.6 to one machine which given a table as in Example 2.5 with each ni ≤ mand (w)2 ≤ n looks up the (w)2-th entry and converts that entry to a binaryrepresentation w1 replacing w, then looks up the (w1)2-th entry, and so on.

3 Basic theory of computability

In this section we deal with four basic issues. First we show that not onlycan we (in principle) simulate a computation of a Turing machine using, say,pencil and paper, but a Turing machine can do it. Secondly we show thatthere are sets which cannot be decided by any Turing machine. In particular,there is no Turing machine which given another Turing machine M as inputcan decide if M stops or not. In this precise sense the termination or haltingproblem is undecidable. Then we discuss semicomputable sets and showthat there are semicomputable sets which in some sense include all othersemicomputable sets. Such sets are said to be complete semicomputable

9

sets. Finally we briefly consider non-deterministic Turing machines. Thesenotions have analouges which are important in the theory of complexity thatwe meet in Section 5.

3.1 Universal Turing machines

In the fundamental paper [12] Turing proved that there is a Turing machineU which given another Turing machine M as input along with an input wfor M simulates the computation of M on w. Such machines are calleduniversal Turing machines. Put in more modern language, Turing showedthe existence of a general purpose computer. Note the important intellectualleap of considering machines as data. The existence of universal Turingmachines is one of the strongest arguments for the validity of the Church-Turing thesis.

From Definition 2.1 we see that a Turing machine is a finite object.Thus each Turing machine can be coded in an effective way as a word oversome alphabet. Similarly, the content of the tape at a particular step in acomputation is a finite word (ignoring blanks to the right and left). Thuswe may code all Turing machines and all tapes (up to isomorphism) aswords over some fixed alphabet. These codings, or Godelisations (afterKurt Godel), should be effective in a natural way. Given a code for a Turingmachine you should be able to write down the machine (up to isomorphism)as in Definition 2.1. And from a description as in Definition 2.1 (includingthe graph of δ) you should be able to write down a code for it.

We will not be more precise about the requirements of our effectivecodings. We are, after all, familiar with the fact that all computer programsand data are coded by zeroes and ones.

A universal Turing machine will operate on codes for Turing machinesand codes for input tapes. Let

M 7→ dM and I 7→ tI

be effective codings of Turing machines and tape contents, respectively.

Definition 3.1. A Turing machine U is a universal machine if given tIdM

as input, where dM is a code for a Turing machine M and tI is a code foran input tape I, then

(i) U halts if and only if M with input I halts, and

(ii) if U halts then the tape content is tOdM where tO is a code for theoutput tape of the machine M from the input I.

Theorem 3.2. There exists a universal machine.

10

The particular construction of a universal machine we present below isdue to Stal Aanderaa. In Scandinavian that machine is therefore calledStalmaskinen.

In our construction we restrict attention to Turing machines satisfyingthe following conditions:

• Each Turing machine is normal with tape infinite only to the left.

• The alphabet of each Turing machine is {0, 1}.

• Each Turing machine has only one passive state, namely h.

It is a fact that each Turing machine can be simulated by a machinesatisfying the conditions above so our restriction is no loss of generality. (Ofcourse, the universal machine we construct will for simplicity not satisfy thesecond condition.)

From Definition 2.2 we see that the machine configuration for a machineM (written as a word rather than a triple) changes in one computation stepfrom waqbv to either wpacv or wacpv depending on the value δ(q, b). IfM is normal then only one direction is possible for each p. We see that wand v are unchanged, i.e. only aqb changes. We call aqb the window of theconfiguration.

The idea of our construction is to code a Turing machine M by the wayit changes the window in one step. There are only finitely many possiblewindows for M . Thus we code M as a finite table of windows. A window willbe regarded as a number i when converted to a binary representation. Theith entry in the table will contain a unary representation of a new numberwhich after conversion to a 5-ary representation is the new window. Theuniversal machine will perform table look up, conversion to new window,table look up, conversion to new window, and so on, as in Examples 2.5 and2.6, until the window contains the halting state.

We let Σ = {0, 1, a, b, A, B, c, d, x, y, h} be the alphabet of our universalmachine. For simplicity we identify blank ] with 0. Given a Turing machineM , we code the states of M as words over {A,B, h}. The machine M iscoded as a table dM as in Example 2.5. A configuration is coded as a word(just to the left of the table) from

{0, 1}∗{0, 1}{A,B, h}{A,B}∗{a, b}{a, b}∗ .

Here a is regarded as another name for 0 and b for 1. The window is thena word from {0, 1}{A,B, h}{A,B}∗{a, b}.

For tI , the code for the input tape to M , we use the input configurationdescribed above. Thus the input to the universal machine is tIdM . Weassume that the reading head of the universal machine starts on the rightmost symbol of its input.

The detailed strategy is as follows (see Figure 6 below):

11

(i) Consider the content of the window as a binary representation of anumber, where 0 = A = a and 1 = B = b. The value is the index ofthe entry in dM which codes the change in the window. (States q0 andq1.)

(ii) Locate the entry as in Example 2.5. (States q2 and q3.)

(iii) Convert the unary representation in the found entry to a 5-ary repre-sentation in the window as in Example 2.6, where 0 is represented byB, 1 is represented by A, 2 is represented by 0, 3 is represented by 1,and 4 is represented by h. (States q4 and q5.)

(iv) Restore the table, change 0 to a and 1 to b to the right of the code forstates. (State q6.)

The Steel machine is given in Figure 6. Note that the number of statesof our universal machine is surprisingly small!

The Steel Machine(Stålmaskinen)

Lq0

Lq3

Rq1

Rq2

A

B

A

B

Hy

A

d

B

0 1

ab

c

d

x

B

x h

A

Rq4

Lq5

B

h

h1

0

A

d

10A

B

c

0 1

Lq6

x

c

x

ab

y01

A B h

H

Figure 6

12

3.2 The halting problem

In the previous section we showed how Turing machines can be coded asand identified with words over a finite alphabet. This means that Turingmachines can ”talk about” Turing machines. The universal machine is anexample of such a machine.

A natural question, which one would have liked to have answered posi-tively, is the following. Is there is a Turing machine D which given anotherTuring machine M as input can decide whether M will halt or not? Put inother words, is the termination or halting problem for computer programsdecidable?

With some thought it would be unreasonable if this were the case. Con-sider Fermat’s last theorem. It states that for each n ≥ 3, the equation

xn + yn = zn

has no nontrivial solution over the integers. It is a simple matter to constructa Turing machine M which searches through all tuples (n, x, y, z) for n ≥ 3and not all x, y, z = 0, and checks whether or not xn + yn = zn. Themachine terminates when and only when such a tuple is found. If therewere a machine D solving the termination problem then we could run iton M . If D says that M terminates then Fermat would be wrong. IfD says that M does not terminate then Fermat would be right. Difficultmathematical questions, like that of Fermat, could in this way be decidedmechanically. Of course, we now know, thanks to Andrew Wiles, that Mwill not terminate. But it took 350 years and a tremendous amount ofmathematical development to find a proof of that.

Thus it is reasonable to believe that such a machine does not exist. Nowwe will prove it.

Let Σ be an alphabet and let A ⊆ Σ∗. The characteristic function of A

is the function cA: Σ∗ → {True,False} defined by

CA(w) = True ⇐⇒ w ∈ A.

Definition 3.3. A set A ⊆ Σ∗

is decidable or computable if its characteristicfunction is computable.

From set theoretic principles we immediately see that there are subsetsA ⊆ Σ

∗which are not computable. Σ

∗is infinite so the power set of Σ

∗,

the set of all subsets of Σ∗, is uncountable. On the other hand, each Turing

machine can be coded by a word in Σ∗

so there are only countably manyTuring machines. Therefore almost all subsets of Σ

∗are not computable

or decidable. On the other hand we are not that interested in almost allsubsets. We are interested in important sets. These are at least in somesense definable. One such set or problem is the halting problem alluded toabove.

13

Let us fix an effective coding of Turing machines M 7→ dM .

The Halting Problem Is there a Turing machine D which given as inputwdM stops with output T if M with input w stops, and stops with outputF if M with input w does not stop?

Theorem 3.4. There is no Turing machine which solves the halting prob-lem.

Proof. Suppose, to obtain a contradiction, that Turing machine D solves thehalting problem for each pair (M,w), where M varies over Turing machinesand w over inputs. Then, in particular, D solves the halting problem foreach pair (M,dM ).

Let E be a Turing machine which does the following:

• E copies the input w so that the tape contains ww.

• Then D takes over the computation.

Thus we connect in series a copying machine and D. Now we give E theinput dM for some machine M . Then E stops with output T if M withinput dM stops. And E stops with output F if M with input dM does notstop.

Now we modify E to obtain E∗ as follows. We alter the definition ofthe δ function for E at the point where E writes T and stops, by replacingthese with a state providing an infinite loop. Thus we have the following:

(E∗, dM ) stops ⇐⇒ (E, dM ) stops with output F⇐⇒ (M,dM ) does not stop.

Now we let M = E∗. Then

(E∗, dE∗) stops ⇐⇒ (E∗, dE∗) does not stop.

This contradiction proves the theorem.

There are many other undecidability results that can be derived fromthe one above. For example, there is no Turing machine which can decidethe halting problem for machines with blank input tapes. (Why?) Neither isit decidable whether a Turing machine terminates for some inputs or termi-nates for all inputs. Interestingly, the latter is strictly “more” undecidablethan the former in a precise sense, i.e. there are degrees of undecidability ordifficulty to compute. (Intuitively, in the former case you only have to findone input that terminates while in the latter you have to verify terminationfor all inputs.)

14

3.3 Semicomputable sets and completeness

Consider the universal machine U constructed in the proof of Theorem 3.2.We can easily modify U in such a way that U does not terminate whenthe input is not on the form wdM . Assuming this, then U stops or haltsprecisely for inputs wdM such that M with input w stops. This set is anexample of a semicomputable set or a halting set.

Definition 3.5. Let Σ be an alphabet. A set A ⊆ Σ∗

is semicomputable ifthere is a Turing machine M such that M with input w stops if and only ifw ∈ A.

For a given machine M we let [M ] be the function computed by M .Thus for each w ∈ Σ

∗,

[M ](w) ={

z if M with input w terminates with output z↑ if M with input w does not terminate.

Thus a set A ⊆ Σ∗

is semicomputable if and only if there is a Turingmachine M such that

A = {w ∈ Σ∗

: [M ](w)↓}.

Semicomputable sets are sometimes called accepting sets in that the set isprecisely the set of words accepted by a Turing machine in the sense that themachine stops. A partial function f is computable if and only if f = [M ] forsome Turing machine M . So semicomputable sets are precisely the domainsof partial computable functions.

Theorem 3.6. There is a semicomputable set A ⊆ Σ∗

which is not com-putable.

Proof. The halting set {wdM : (M,w) terminates} is semicomputable aswitnessed by a universal machine, but the set is not computable by Theorem3.4.

In the proofs below we will be (even) more informal when describingalgorithms. We rely on the Church-Turing thesis and trust that the scepticalreader is able to translate our algorithms to Turing machines.

Theorem 3.7. A set A ⊆ Σ∗

is computable if, and only if, A and thecomplement A = Σ

∗ −A are semicomputable.

Proof. Assume A is computable. Let M be a Turing machine decidingA. Alter M by introducing a loop at its F exit. The altered machinestops precisely on A, i.e. A is semicomputable. A is also computable, byinterchanging the T and F exits, so A is also semicomputable.

15

Now assume both A and A are semicomputable, witnessed by the ma-chines M and M , respectively. We constuct a new machine M as follows.The machines M and M are to be run in parallel. (Say we first run M for10 steps then run M for 10 steps, then run M for 10 steps and so on. This isdoable by adding a “clock”, i.e. a machine counting steps.) If M terminatesthen M will write T and stop. If M terminates then M will write F andstop. Let w ∈ Σ

∗be given as input to M . Then precisely one of M and M

will terminate depending on whether w is in A or not, so M computes thecharacteristic function of A.

We shall now define two notions which are equivalent to semicomputable.One should note, however, that when considering computability in othersettings these notions may diverge.

Definition 3.8. A set A ⊆ Σ∗is computably enumerable (c.e.) or recursively

enumerable (r.e.) if there is a Turing machine M which starting with anempty tape lists all the members of the set A (or if A = ∅).

The machine will normally not stop and cannot stop if A is infinite.After an infinite number of steps in the latter case the tape will contain aninfinite list w0, w1, w2, . . ., each word separated by a blank ]. Then

A = {wn : n ∈ N}.

Note that the words are not necessarily generated in an increasing order(with respect to the lexicographic ordering of Σ

∗). If they are then the set

is computable (why?). We also allow repetitions of words in the list.The notion of computably enumerable is very natural, perhaps more

natural than the notion of decidable. For example the natural numbers areobtained in this way. We start with 0, then we add 1; then we add 1; etc.Another example is the set of words over an alphabet.

Theorem 3.9. A set A ⊆ Σ∗

is semicomputable if, and only if, A is com-putably enumerable (or r.e.).

Proof. Suppose A is r.e., say A is generated by a Turing machine M . LetM ′ be a machine which given w as input generates A using M . After a wordw′ is generated, M ′ checks if w = w′. If yes, M ′ stops. If no, then M ′ usesM to generate the next word. And so on. Then clearly A = dom([M ′]), soA is semicomputable.

The converse involves the notion of dove-tailing. Let A = dom([M ]).The machine M ′ we construct to generate A will do the following. Usinga clock the machine M ′ will first enumerate the first hundred words in thelexicographic ordering of Σ

∗and then determine in turn for each such word

w whether or not M with input w terminates in less than hundred steps.

16

Those words for which this holds are then generated. Now we iterate theprocedure, but we consider the first two hundred words (so we again considerthe first hundred) and we run M on each of them and check for which onesM terminates in less than 200 steps. Then the words for which this holdsare generated. The procedure is iterated infinitely many times, each timethe number of words and steps to be checked are incremented by 100 (orwhatever positive number you want). Clearly M ′ generates A, though ofcourse each word in A appears infinitely often.

Often one wants to consider functions of several arguments or relationsof arbitrary arity. This is easily incorporated into the Turing machinery by,for example, introducing “,” as a new symbol into the alphabet (analogouswith ]). Thus f : Σ

∗ → Σ∗

is computable if there is a machine M whichgiven the word w, v as input computes f(w, v). We consider decidability ofrelations of arbitrary arity in an analogous manner.

Let M be a Turing machine. Then we define a binary relation TM onΣ∗

(depending on M) by

TM (w, z) ⇐⇒ M with input w terminates in ≤ |z| steps.

Clearly TM is a computable relation. Given w and z we simply run M on wwhile at the same time, i.e. in parallel, run a clock to if necessary terminatethe procedure when M has performed |z| steps.

A set A ⊆ Σ∗

is said to be Σ1-definable if there is a binary computablerelation B on Σ

∗such that for each w ∈ Σ

∗,

w ∈ A ⇐⇒ ∃zB(w, z).

Theorem 3.10. A set A ⊆ Σ∗

is Σ1-definable if, and only if, A is semi-computable.

Proof. Suppose A is Σ1-definable using B. Let M be a machine decidingB. Let M ′ be the machine which when given w systematically generates allwords z and when a z is generated checks if B(w, z) holds. If it holds thenM ′ stops. If it does not hold then M ′ generates the next word, and so on.Clearly A = dom([M ′]), i.e. A is semicomputable.

Conversely, suppose A is semicomputable, say A = dom([M ]). Then

w ∈ A ⇐⇒ ∃zTM (w, z),

so A is Σ1-definable.

Remark 3.11. It is the case that TM is uniform in M . This means thatthere is a machine M which given M,w, z as input decides TM (w, z). Fur-thermore we could have included in z not only the number of steps computed

17

but also the value of the computation which then could easily be retrieved.In this way we obtain a computable predicate

T (M,w, z)

which is a version of Kleene’s famous T-predicate.

We know that there are semicomputable sets that are not computable. Isit the case that some semicomputable non-computable sets are more difficultto compute than others? To answer such a question we have to decide onsome measure or reduction saying that one set is at least as difficult tocompute as some other set. There are several such reductions that have beenstudied. Here we will only mention one such reduction since this reductionwill be important later when considering complexity classes.

Definition 3.12. Let A and B be subsets of Σ∗. Then A is said to be m-

reducible to B, denoted by A ≤m B, if there is a total computable functionf : Σ

∗ → Σ∗

such that for each w ∈ Σ∗,

w ∈ A ⇐⇒ f(w) ∈ B.

Theorem 3.13. Suppose A ≤m B. If B is computable then A is computable.And if B is r.e. then A is r.e.

Proof. Let f be the witness to A ≤m B. Given w ∈ Σ∗

compute f(w) andcheck if f(w) ∈ B. If B is computable then this procedure is computable. IfB is semicomputable using machine M then so is A, composing a machinefor f with M .

Using m-reducibility one sees that there is a most difficult r.e. set in thesense that all other r.e. sets are reducible to it.

Definition 3.14. An r.e. set A is a complete r.e. set if for every r.e. set B,B ≤m A.

Theorem 3.15. There is a complete r.e. set.

Proof. Let K0 = {wdM : w ∈ Σ∗,M a TM, and [M ](w)↓}. Note that K0

is the halting set used in the proof of Theorem 3.4. Thus K0 is r.e. Let Abe an r.e. set and let M be a Turing machine such that A = dom([M ]).The function f defined by f(w) = wdM is clearly total and computable.A machine computing f(w) simply writes the constant word dM after theinput w. This is easily coded into the finite set of states of the machine. Itfollows that

w ∈ A ⇐⇒ f(w) = wdM ∈ K0.

A natural question to ask is if every r.e. set is either computable orcomplete. The answer is no, as shown by Post [10]. This is one startingpoint into the exciting world of computability theory which we cannot pursuefurther in this course.

18

3.4 Nondeterministic Turing machines

The notion of a nondeterministic Turing machine will be important in thestudy of complexity classes. A nondeterministic machine is one which ateach step in a computation has several choices of action. Given such amachine M we say that M accepts a word w if there is a sequence of stepsfor which M terminates given w as input. Thus we can think of such amachine as having an extra instruction, namely guess which of the finitelymany actions to take at a given step. A word is then accepted if thereexists a sequence of correct guesses making the machine terminate. Notethe existential quantifier there exists!

Definition 3.16. A nondeterministic Turing machine is a five-tuple M =(Q,Σ, δ, q0, F ) where Q,Σ, q0 and F are as in Definition 2.1 but where

δ: (Q− F )× Σ] → ℘(Q× Σ] × {R,L})

where ℘ is the power set operator.

The relation ` on configurations now changes as follows. We say that

(wa, q, bv) ` (wac, p, v) if (p, c, R) ∈ δ(q, b)

and similarly

(wa, q, bv) ` (w, p, acv) if (p, c, L) ∈ δ(q, b).

Thus ` is no longer deterministic in that several configurations can follow

from one configuration. Now we let∗` be the transitive closure of `.

Definition 3.17. Let M be a nondeterministic Turing machine. Then thelanguage accepted by M is

L(M) = {w ∈ Σ∗

: ∃(x, p, y)((ε, q0, w)∗`(x, p, y) and p ∈ F )}.

Informally, w ∈ L(M) if there is at least one computation of M withinput w which terminates. Each r.e. set is L(M) for some deterministicTuring machine. We now show that we do not get any new sets by going tonondeterministic machines.

Theorem 3.18. A language is accepted by a nondeterministic Turing ma-chine if, and only if, it is accepted by a deterministic Turing machine.

Proof. (Sketch) Assume A = L(M1) where M1 is a nondeterministic Turingmachine. Let n be a number so that the cardinality of each set δ(q, a) is≤ n. Construct a machine M2 which systematically generates all wordsover {1, 2, . . . , n}. We now construct a deterministic machine M as follows.Suppose M2 has generated a sequence i1, i2, . . . , ik. Then run M1 for (atmost) k steps in the following way. At step j, choose the ij-th alternativefrom the δ function of M1. If a word w is accepted by M1, then eventuallyM2 will generate a sequence describing that computation and hence M willaccept w.

19

4 Complexity

The notion of an algorithm is arguably one of the most fundamental conceptsin mathematics. The theory of algorithms, also called computability theoryor recursion theory, is a beautiful and by now well developed theory withimplications in many areas of mathematics, logic and computing. However,computability theory is a theory of what can be done in principle, i.e. withoutthe concern of the amount of resources to be used.

With the appearance of real physical computers, the question of resourcesbecomes important. After all, a real computer is not a Turing machine. Ithas an a priori limitation to its storage capacity and there is also an a priorilower bound for the time duration of one step in a computation. Complexitytheory is the general theory of algorithms where the concerns of space andtime resources are included.

In the sequel we will often consider multi-tape Turing machines, a moregeneral variant of a Turing machine than that of the previous sections. Amulti-tape Turing machine will have several tapes, a reading head for eachtape and the ability to in one step write a symbol on each tape, move eachreading head one square to the left or one square to the right or remainwhere it was, all depending on the state and the contents of the squaresunder all the reading heads. The tapes will be infinite in only one direction,to the right. Two of the tapes will be special. One tape will be the inputtape. It is a read-only tape, i.e. the machine can read symbols on the tapebut never alter the contents of the tape. Another tape is the output tape.This is a write-only tape, i.e. the machine can only be stationary or move tothe right on that tape. Remaining tapes are called work tapes. Suppose sucha machine is to compute a partial function. Then the argument is writtenon the input tape while remaining tapes are blank. Each reading head isplaced over the left most square of its tape. When (or if) the machine stops,the content of the output tape is the value of the function.

We leave to the reader the formalisation of multi-tape machines, thefunctions which they compute, and the sets which they accept. It shouldbe clear that no new functions are computed and no new sets are accepted,using this more general model. However the generalisation has a small effecton the efficiency as shown in the next section.

4.1 Basic definitions and results

The resources considered are those of time and space. Time is measured inthe number of steps of a computation and space by the number of squaresused in a computation. These complexity notions are given as a function ofthe size of the input.

Definition 4.1. Let M be a (multi-tape) Turing machine.

20

(i) M has time complexity T (n) if M stops within T (n) steps for eachinput of length n.

(ii) M has space complexity S(n) if at most S(n) squares of the workingtapes are visited during a computation for each input of length n.

We often use the phrases “runs in time T (n)” for having time complexityT (n), and similarly “uses space S(n)” for space complexity S(n). Note thecondition on space complexity that we do not count the input tape. Ifthe input tape would be counted then we would always have S(n) ≥ n.However we will later be interested in situations where the space used forthe computation is log(n).

In the sequel we will only consider total computable functions and de-cidable sets. We say that a set A is accepted by a Turing machine M if Mtakes the value T or F for each input, and w ∈ A ⇐⇒ the value of M is T.A is accepted by a nondeterministic Turing machine M if every computa-tion of M terminates with value T or F, and w ∈ A ⇐⇒ there is (at least)one computation from input w which has the value T. In each case we sayA = L(M).

A first result is that there are arbitrarily complex computable functions.

Proposition 4.2. Let f be a computable function. Then there is a {T,F}-valued computable function F which cannot be computed in time f(n).

Proof. Recall that we may take codes for Turing machines as arguments asin the discussion of universal machines. It is clearly decidable whether aword codes a Turing machine or not. Therefore, for simplicity we write Mfor a code for M and restrict the argument to such codes. Define

F (M) =

T if M stops with output F for input M

in at most f(|M |) steps;F otherwise.

Suppose F is computed by a machine M0 having time complexity f(n).Then

F (M0) = T ⇐⇒ M0 with input M0 stops with output F ⇐⇒ F (M0) = F.

To see that F is computable we use a universal machine and in parallelwe run a “clock” which checks that the simulated machine uses at mostf(|M |) steps.

A similar argument works for the analogous proposition dealing withspace complexity.

Space complexity is insensitive to linear changes. Recall the function

dre = least n ∈ N such that r ≤ n

for r ∈ R. Similarly brc donotes the largest n ≤ r. We will often write, forexample, log(n) when we really mean dlog(n)e.

21

Proposition 4.3. Let c > 0 be a real number and suppose S: N → N is afunction. If A is a set accepted by a Turing machine M with space complexityS(n) then there is a Turing machine M accepting A in space c · S(n).

Proof. (Sketch) Let t ∈ N be a constant to be chosen later. The idea is tocode t symbols from the alphabet of M to one symbol in the alphabet of M .M is a machine which simulates M as follows. A tape for M is divided intoblocks of t squares. Each block is coded as one square for the correspondingtape for M . The states of M codes where in a block the reading heads ofM are located at a particular step.

Clearly the space used by M is ≤ dS(n)t e. Choose t such that tc ≥ 2.

ThendS(n)

te ≤ dc · S(n)

2e ≤ c · S(n).

The same proof holds for nondeterministic machines.In the proof above, the machine M uses as many steps as M . In order to

obtain a similar result for time complexity we need to code finite sequencesof computation steps. Given M we again divide each tape for M into blocksof t squares. In our simulating machine we have a corresponding tape whereeach block is coded as one square. As long as M does not change blocks thenM needs to do nothing. However, if M crosses the boundary of a block, thenM must also perform a step. We have to avoid the problem of a machineM crossing the boundary very often thus forcing M to take many steps. Asolution is to consider regions, a region being also a block to the right anda block to the left. If M leaves a region, which takes at least t steps, thenM only needs to modify its configuration which takes at most 4 steps.

Initially the machine M needs to code the input for M to the tape formatfor M . M reads the input from left to right and codes it onto a work tape.Then M uses the work tape as input tape and the input tape as a worktape. This is linear in the size of the input. Note that here we assume thatM has k tapes for k ≥ 2.

Using this idea one can show that if M performs T (n) steps then Mperforms at most

6kdT (n)t

e+ number of initial steps

which is less than 6kdT (n)t e+ an for some constant a.

Now suppose limn→∞T (n)

n = ∞. Then for each s ∈ N there is ns suchthat for all n ≥ ns,

6kdT (n)t

e+ an ≤ T (n)(6k

t+

a

s).

22

If we now, given c > 0, choose t and s such that 6kt + a

s < c, then M simulatesM in c · T (n) almost everywhere. We have sketched a proof of

Proposition 4.4. Assume limn→∞T (n)

n = ∞ and let c > 0. If a set A isaccepted in time T (n) by a k-tape machine for k ≥ 2, then A is also acceptedin time c · T (n) by a k-tape machine.

The proposition also holds for nondeterministic machines with the sameproof.

Note that if T (n) = c·n then limn→∞T (n)

n = c and hence the propositionis not applicable. However in this case it is possible to go down to complexity(1 + ε)n almost everywhere, for each ε > 0. (Exercise.)

We recall the big O notation. Let f : N → N. The we say g ∈ O(f) ifthere is c > 0 such that g(n) ≤ cf(n) for almost all n. (Almost all meansall but finitely many as does almost everywhere.)

We have seen from the propositions above that complexity is insensitiveto linear changes. Therefore we freely identify complexity S(n) with O(S(n))and T (n) with O(T (n)).

Finally we consider the reduction of a k-tape Turing machine to a 1-tapeTuring machine. The idea in the proof is similar to the ideas above: youcode k tapes to one tape. We leave the details to the reader.

Proposition 4.5. Let A ⊆ Σ∗

be a set.

(i) If A is accepted in space S(n) by a k-tape Turing machine then A isaccepted in space S(n) by a 1-tape machine.

(ii) If A is accepted in time T (n) by a k-tape Turing machine and it isthe case that limn→∞

T (n)n = ∞, then A is accepted in time T 2(n) by

a 1-tape machine.

4.2 Complexity hierarchies

Let f : N → N. Then DTIME(f) is the complexity class consisting of the sets(subsets of Σ

∗for a fixed Σ) accepted by some deterministic Turing machine

running in time f(n). The classes NTIME(f) (nondeterministic machines),DSPACE(f), and NSPACE(f) are defined similarly.

Here is a first proposition showing some relations between complexityclasses.

Proposition 4.6. Let f : N → N.

(i) DTIME(f) ⊆ DSPACE(f).

(ii) NTIME(f) ⊆ DSPACE(f).

(iii) DSPACE(f) ⊆⋃∞

c=1DTIME(cf ) if f(n) ≥ log(n).

23

(iv) NTIME(f) ⊆⋃∞

c=1DTIME(cf ).

(v) NSPACE(f) ⊆DSPACE(2f ).

Proof. We only prove (ii). The proof is similar to that of Theorem 3.18 withthe difference that we have to be careful about the space used. Let M bea nondeterministic k-tape Turing machine which runs in time f(n). Let mbe maximum of the cardinalities |δ(q,~a)| for q ∈ Q and ~a ∈ Σk. Let M be amachine with k + 1 tapes. On one of the work tapes M will systematicallygenerate words in {1, 2, . . . ,m}∗, one word at the time. When a word isgenerated then M simulates a computation of M on the other tapes, wherethe specific computation chosen is determined by the generated word. IfM accepts for this computation then M accepts and stops. If M does notaccept then M generates a new word, replacing the old one on the tape,and simulates M according to the just generated word. As before, M willaccept if and only if M accepts. Note that a computation of M will have atmost f(n) steps and hence it suffices for M to generate words of length f(n).Furthermore, the simulation on the other tapes can for the same reason useO(f(n)) space.

It turns out that (ii), (iii) and (iv) are best possible, while (v) can beconsiderably improved to f2 for reasonable f and (i) can be improved tof/ log f .

What does reasonable f mean? Surprisingly, just to be computable isnot enough. Borodin [2] has shown that given any computable function h(and h can be chosen to grow extremely fast) there is a computable functionf such that

DTIME(f(n)) = DTIME(h(f(n))).

Thus there is a “gap” between f(n) and h(f(n)) containing no new problems.A key often used in the simulation proofs we have given above is the

existence of a clock. That is we can run a clock in parallel with the machineM to be simulated counting the number of steps taken by M or the numberof squares used by M such that the simulation can stop when the clock hasrun out. Now we require more of the clocks.

Definition 4.7. Let f : N → N be a computable function.

(i) f is time constructible if there is a Turing machine which for eachinput of length n stops after exactly f(n) steps.

(ii) f is space constructible if there is a Turing machine which for eachinput w with |w| = n places a special symbol on the f(n)th square ona designated tape and uses at most (and hence exactly) space f(n).

It is the time constructible and space constructible functions which arethe reasonable functions.

24

It is a fairly easy exercise to show that polynomials and the exponentialfunction 2n are both space and time constructible and that dlog(n)e is spaceconstructible.

We have already seen from Proposition 4.2 that there are arbitrarilydifficult computable functions with respect to time complexity. The same istrue of space complexity with a similar proof. Now we want to consider amore delicate problem. How much larger must T2(n) be in relation to T1(n)in order for there to be a set accepted in T2(n) time but not in T1(n) time?

In order to prove such a hierarchy result it is necessary to analyse thecost of reducing a k-tape machine to a 2-tape machine. We omit the proofof the following lemma.

Lemma 4.8. For each Turing machine M running in time T (n) there is a2-tape Turing machine M and a constant c (depending on M) such that Msimulates M and M runs in time c · T (n) · log(T (n)).

Theorem 4.9. Let T2(n) be time constructible and assume that

limn→∞

T2(n)T1(n) · log(T1(n))

= ∞.

Then there is a set A ∈ DTIME(T2(n))−DTIME(T1(n)).

Proof. (Sketch) Code effectively all 2-tape Turing machines in such a waythat if dM is a code for a machine M then dMan is also a code for M , foreach n. Thus there is a universal machine which given dMan can efficientlysimulate M using dM .

We now define a machine M . Given input w, M will do the following:

• M clocks itself and turns itself off after T2(|w|) steps. (Here we use thatT2 is time constructible.) Thus we know that L(M) ∈ DTIME(T2(n)).

• M checks that the input w is on the form dMan. If not then M rejects.

• M simulates the 2-tape machine Mw with input w using an efficientuniversal machine. M accepts w if and only if the simulation of Mw

on w is completed and Mw does not accept w.

Let M be a k-tape machine running in time T1(n). Then M can besimulated by a 2-tape Turing machine M running in time c·T1(n)·log(T1(n))for some c depending on M . Choose code w such that M = Mw, M cansimulate Mw on w quickly (i.e. in < T2(|w|) steps), and

c · T1(|w|) · log(T1(|w|)) ≤ T2(|w|).

Then M will have enough time to simulate M on w. Thus

w ∈ L(M) ⇐⇒ w /∈ L(M).

This proves that L(M) /∈ DTIME(T1(n)).

25

Thus, for example, DTIME(nm) ⊂ DTIME(nm+1), and DTIME(2n) ⊂DTIME(n2 · 2n), where ⊂ denotes proper subset.

A similar, but sharper, hierarchy theorem holds for space complexity. Weleave the proof as an exercise. (The lemma to know is that the simulationof an S(n) machine M runs in space c ·S(n) where c is dependent on M , forS(n) ≥ log(n)).

Theorem 4.10. Let S2(n) be space constructible such that S2(n) ≥ log(n)and assume that

limn→∞

S2(n)S1(n)

= ∞.

Then there is a set A ∈ DSPACE(S2(n))−DSPACE(S1(n)).

5 Feasible computations

It is possible to give a reasonable and robust mathematical definition of theclass of sets and problems that can be decided or solved in practice? In otherwords is there a practical Church-Turing thesis? Cobham [3] and Edmonds[5, 6] proposed that the class of sets decidable in practice is the class whichcan be decided by Turing machines running in polynomial time. However,they were also aware that certain very natural problems were probably notin this class.

Example 5.1. Integer linear programmingSuppose given an m×n matrix A and a m×1 matrix b, is there x such thatAx ≥ b? All the matrices are over the integers.

It is still unknown whether the above problem can be solved in polyno-mial time though it is very unlikely.

Consider the above problem where you allow the solution to be over therationals. There is a well-known rather old algorithm to solve that problem,namely the simplex algorithm. It has proved very efficient in practice but itdoes not run in polynomial time. In 1979 Khachiyan [8] found a polynomialalgorithm for linear programming. The result made the newspapers and itcertainly was an important break-through. However, interestingly enoughthe polynomial algorithm produced was usually slower than the simplexalgorithm! (Later the polynomial algorithm has been improved to be asefficient.) The moral of the story is that it may be preferable to use an algo-rithm which works well in most cases even though it may be very inefficientin a few cases. However, in our theory here we require our algorithms towork well for all inputs.

We now define various classes which may be said to contain the (moreor less) feasible problems.

Definition 5.2. (i) P =⋃∞

k=1 DTIME(nk).

26

(ii) NP =⋃∞

k=1 NTIME(nk).

(iii) L = DSPACE(log(n)).

(iv) NL = NSPACE(log(n)).

(v) PSPACE =⋃∞

k=1 DSPACE(nk).

Perhaps one would expect a nondeterministic version of (v). However,due to Theorem 5.10 below, that class is exactly PSPACE .

The most important classes are P and NP. It should be noted that theyare independent of the machine model chosen. For example, if a one tapeTuring machine is chosen as a computation model then the time to computeon it instead of on a k-tape machine is increased by at most a power of 2.

Clearly P ⊆ NP. A useful way to picture NP is given in the followingtheorem.

Theorem 5.3. Let A ⊆ Σ∗. Then A ∈ NP if, and only if, there is a binary

relation B ∈ P and a polynomial p(n) such that

w ∈ A ⇐⇒ ∃v(|v| ≤ p(|w|))B(w, v).

We will not prove the theorem but its truth should be intuitively clear, atleast in one direction. Suppose the condition holds. Then a nondeterministicTuring machine M with input w will:

• guess (i.e. nondeterministically generate) a v such that |v[≤ p(|w|),

• then compute deterministically whether or not B(w, v).

Thus the computation is divided into two parts; a guess and then a fastdeterministic computation. A word w is accepted precisely when there isa correct guess. Note that the size of a guess is limited by a polynomialin |w|, so to generate a guess will take time polynomial in |w|. Since B isdecidable in time polynomial in |wv|, and hence in 2|w|, the machine willrun in polynomial time.

If A satisfies the condition in Theorem 5.3 we say A is Σp1 over P. Note

the analogy between computable and P, and semicomputable and NP.Another analogue in logic to illustrate the difference between NP and

P is to

• effectively find a proof;

• effectively verify a proof.

Consider Integer linear programming of Example 5.1. To show that theproblem is in NP assume we are given A and b. Then simply guess an xand decide whether Ax ≥ b. We must of course know that if such an x

27

exists then its size must be polynomial in A and b and this is well-known.Furthermore matrix multiplication is quick. (There are algorithms runningin time na where 2 < a < 3.)

Here are a few more problems which are easily shown to be in NP.

Example 5.4. CLIQUEGiven an undirected graph G and a number k. Is there a clique of size ≥ k?(A clique is a subgraph where every node is connected to every other node.)(G, k) is accepted if and only if there is such a clique.

Example 5.5. SATGiven a Boolean formula F in conjunctive normal form. Is there an eval-uation which makes F true? (That is, is the formula F satisfiable?) F isaccepted if and only if F is satisfiable.

Note that each of these problems has an existential quantifier in itsformulation. Thus to show that these problems are in NP it suffices toshow that if there is a witness to the existential quantifier then there is awitness with length limited by a polynomial; and that the verification isquick, i.e. polynomial. This should be clear in both the examples above.

We shall now determine the known relations between the classes intro-duced in Definition 5.2. For this we first consider the following problem.

Example 5.6. REACHABILITYGiven a directed graph G with two nodes s and t as inputs, accept (G, s, t)if and only if there is a directed path from s to t.

We provide an algorithm using a Turing machine with two work tapesR and W . Here is the algorithm.

Write s on R.

Write all v on W for which (s, v) ∈ G, i.e. there is an edge froms to v.

While W is not empty do

Move the first element w on W to R. Check G to writeevery node u on W for which there is an edge from wto u and such that u is not already on R or W .

If t is on R then accept, otherwise do not accept.

Note that R will at any time only contain nodes which are reachable froms and at termination R will contain all nodes reachable from s. We leave itas an exercise to show that the machine runs in time O(n2) in the numberof nodes of G. Hence the problem REACHABILITY is in P.

28

We now show that REACHABILITY is also in NL. Here is a non-deterministic Turing machine for the problem. The machine will have oneinput tape and two work tapes, tape H and tape Counter. Assume the inputis given as the list of pairs of nodes, each pair (u, v) denoting an edge fromu to v, followed by the nodes s and t. Here is the algorithm:

Check input to find the number of nodes n. Initialise Counter tohave size log(n) in order for M to stop after n iterations below.

Set H = s (i.e. write s on tape H and set Counter to 1).

Do for i = 1, 2, . . . , n. (Using Counter, where n is the number ofnodes of G.)

Let H contain u. If u = t then accept and stop.If u 6= t and there is no edge (u, v) then reject andstop.If u 6= t and there is edge (u, v) then choose nondeter-ministically some such v and replace u by v on H.Update Counter by 1.

Stop and reject.

Clearly the machine solves REACHABILITY. Does it run in log(n) space?The counter is no problem since using binary representation it uses log(n)squares. In order to restrict H to log(n) we represent the nodes as 1, 2, . . . , nwritten in binary notation. Note that H only contains one node at any giventime. Thus the space for H is also log(n). We conclude that REACHA-BILITY is in NL.

Using REACHABILITY we are now able to prove the following im-portant theorem.

Theorem 5.7. Suppose S(n) ≥ log(n) is space constructible. Then

NSPACE(S(n)) ⊆ DTIME(2O(S(n))).

Corollary 5.8. NL ⊆ P.

Proof. Let S(n) = log(n).

Proof. (Theorem 5.7)Suppose A ∈ NSPACE(S(n)). Let M be a nondeterministic Turing machinewhich accepts A in space S(n). Assume without loss of generality that Mhas the following properties:

• M has one work tape.

• The alphabet has 2 symbols.

29

• M has Q states.

Then, given input x, there are at most

O(|x| ·Q · S(|x|) · 2S(|x|))

configurations. (|x| gives the number of positions for the reading head on theinput, and S(|x|) for the work tape. There are 2S(|x|) possible work tapes.)We may also assume that there is exactly one accepting configuration, givenx.

Let Ix be the starting configuration for input x and let Jx be the uniqueaccepting configuration. Given input x let GM,x be the following directedgraph.

• Nodes: all the possible configurations.

• Edges: (I, J) is an edge if I ` J for configurations I and J . Recall that` is a one step computation by M from configuration I to configurationJ .

Note that x is accepted by M if and only if there is a path from Ix to Jx.Therefore we now consider the REACHABILITY problem for directedgraphs (GM,x, Ix, Jx).

Let N be the following deterministic Turing machine. Given input x,

• N generates (GM,x, Ix, Jx) onto a tape. (N needs to know S(|x|) inorder to generate configurations of correct length, but this is possiblesince S(n) is space constructible.)

• N checks if there is a path from Ix to Jx.

• N accepts if and only if there is such a path.

Thus N accepts A.The generation of (GM,x, Ix, Jx) takes time 2O(S(|x|)). To check REACH-

ABILITY is according to Example 5.6 polynomial time in input and hencetakes time 2O(S(|x|)). It follows that N runs in time 2O(S(|x|)).

As we have seen above, REACHABILITY is in NSPACE(log(n)). Bya clever algorithm Savitch [11] proved

Theorem 5.9. REACHABILITY is in DSPACE(log2(n)).

We will omit the proof. However, the result can be used to show thefollowing important theorem.

Theorem 5.10. (Savitch’s Theorem) Let S(n) ≥ log(n) be space con-structible. Then

NSPACE(S(n)) ⊆ DSPACE(S2(n)).

30

It follows that nondeterministic PSPACE is the same as deterministicPSPACE .

We now have all the known relations between the five classes we haveintroduced.

Theorem 5.11. (i) L ⊆ NL ⊆ P ⊆ NP ⊆ PSPACE.

(ii) NL is a proper subset of PSPACE.

Proof. (i) The only inclusions which are not immediate are NL ⊆ P andNP ⊆ PSPACE . The first of these is Corollary 5.8 and the second ispart (ii) of Proposition 4.6.

(ii) limn→∞n

log2(n)= ∞. Therefore, by Theorem 4.10, DSPACE(log2(n))

is a proper subset of DSPACE(n) ⊆ PSPACE . ButNL ⊆ DSPACE(log2(n))by Theorem 5.10, so we are done.

The amazing part is that nothing more is known despite much effort bymany researchers.

6 The Cook-Levin theorem and related results

In this final section we concentrate our attention to the classes P and NP.We show the famous theorem due to Cook [4] and Levin [9] independently,stating that there are “most difficult” natural sets in NP.

The notion of “most difficult” we choose is analogous to that of beinga complete semicomputable set as described in Section 3.3. We modify thedefinition of ≤m as follows.

Definition 6.1. Let A ⊆ Σ∗

and B ⊆ ∆∗.

(i) A is polynomially reducible to B, A ≤p B, if there is a total functionf : Σ

∗ → ∆∗

computable by a Turing machine running in polynomialtime, such that for all x ∈ Σ

∗,

x ∈ A ⇐⇒ f(x) ∈ B.

(ii) A is logspace reducible to B, A ≤l B, if there is a total functionf : Σ

∗ → ∆∗

computable by a Turing machine using log space, suchthat for all x ∈ Σ

∗,

x ∈ A ⇐⇒ f(x) ∈ B.

It is an easy exercise to show that A ≤l B =⇒ A ≤p B. But it isstill unknown if they are equal. A reason for considering ≤l is that in P wecannot separate sets using ≤p. Thus in an attempt to distinguish NL from

31

P we are forced to use ≤l. Below we will only consider ≤p. We leave as achallenge to the reader to verify that all ≤p reductions we perform in thissection can be done in log space.

Proposition 6.2. (i) ≤p is a preorder, i.e. it is reflexive and transitive.

(ii) A ≤p B if, and only if, Σ∗ −A ≤p ∆

∗ −B.

(iii) If B ∈ P and A ≤p B then A ∈ P.

(iv) If B ∈ NP and A ≤p B then A ∈ NP.

Proof. We will only prove (i). Reflexivity is trivial so we show transitivity.Suppose A ≤p B and B ≤p C via f1 an f2 respectively, where f1 is computedin time O(p1(n)) and f2 in time O(p2(n)). Let g = f2 ◦ f1. Then x ∈ A ⇐⇒g(x) ∈ C for each x ∈ Σ

∗. Note that |f1(x)| ≤ p1(|x|) since it requires a

step to write a symbol of f1(x). The function g is computed by a standardcomposition of the machines computing f1 and f2. This machine runs intime

O(p2(p1(|x|))) + p1(|x|) = O(q(|x|))

for some polynomial q.

We now have the corresponding notion of completeness.

Definition 6.3. Let F be a class of subsets of Σ∗.

(i) A ⊆ Σ∗

is F-hard if for each B ∈ F , B ≤p A.

(ii) A ⊆ Σ∗

is F-complete if A ∈ F and A is F-hard.

Theorem 6.4. There is an NP-complete set.

Proof. We assume all nondeterministic Turing machines and alphabets arecoded over some fixed alphabet ({0, 1} will do). Let NPCOMP be thefollowing problem.

(M,x, y) ∈ NPCOMP if and only if M is a code for a nonde-terministic Turing machine, x is a code for input, and there is acomputation by M on input x which accepts x within |y| steps.

Suppose A ∈ NP and let M be a nondeterministic Turing machine whichaccepts A in time p(n) for some polynomial p. Define a function f by

f(x) = (M,x, 1p(|x|)).

The computation of f is trivial. f writes the constant M , then the inputx, and finally writes p(|x|) ones. This is clearly computable in polynomialtime. Furthermore

x ∈ A ⇐⇒ f(x) ∈ NPCOMP

32

for each x ∈ Σ∗

so A ≤p NPCOMP. We have shown that NPCOMP isNP-hard.

It remains to show that NPCOMP is in NP. We construct a machineN in the usual manner. Given input, N checks that it has the form (M,x, y)where M is a code for a nondeterministic Turing machine and x is a code forinput. Then N guesses a word w ∈ {1, 2, . . . , t} such that |w| = |y|, where tis the maximal cardinality of δ(q, a) for the machine M . Finally it simulatesthe computation of M decided by w. N accepts the input precisely whenM accepts input x. Clearly N operates in polynomial time.

It suffices to consider the problem NPCOMP or any otherNP-completeproblem to decide if P = NP.

Theorem 6.5. P = NP if, and only if, NPCOMP ∈ P.

Proof. If A ∈ NP then A ≤p NPCOMP. Hence A ∈ P if NPCOMP ∈ P,by Proposition 6.2.

If P 6= NP then we know that NPCOMP cannot be computed inpractice. But this would perhaps not concern us since NPCOMP is mainlya theoretical device rather than a natural problem. It is a fact that thereare hundreds of “natural” NP-complete problems. The book [7] by Gareyand Johnson contains a long list of such problems.

We now set out to prove the Cook-Levin theorem showing that SAT isNP-complete (see Example 5.5).

A Boolean formula is a formula with variables varying over {T,F} andwith operations ∧ (and), ∨ (or), and ¬ (not). A Boolean formula ϕ issatisfiable if there is an assignment to the variables in ϕ making the formulaTrue. A literal is x or ¬x where x is a variable.

A Boolean formula ϕ is said to be in conjunctive normal form (CNF) if

ϕ =p∧

i=1

(l(i)∨j=1

σij)

where σij is a literal.

Theorem 6.6. SAT is NP-complete.

Proof. We have already observed that SAT is in NP. The relation ≤p istransitive by Proposition 6.2, so it suffices to show NPCOMP ≤p SAT inorder to conclude that SAT is NP-hard. Thus we must define a functionf ,

f : {(M,x, y) : M,x, y ∈ {0, 1}∗} → CNF

computable in polynomial time and such that

(M,x, y) ∈ NPCOMP ⇐⇒ f(M,x, y) ∈ SAT.

33

Here we have assumed that machines and tapes are coded over {0, 1}. Wealso assume that our machines have only one tape which is infinite only tothe right, have only one accepting state, and that 0 serves as blank.

Strategy : Given (M,x, y) we code all possible computations of M withinput x of length ≤ |y| as a Boolean formula in CNF such that there is anaccepting computation for M on x if and only if the corresponding formulais satisfiable.

00

1

1 i

t

|y|time

|y|tape

squares

state attime t

i :th symbol from left at time tIndication whether reading head is oversquare i at time t.

Figure 7

Assume we are given (M,x, y). Let the states of M be {q1, . . . qs}. Weconstruct a Boolean formula with the following variables.

• Ci,j,t for 0 ≤ i, t ≤ |y| and j = 0, 1.(Interpretation: Ci,j,t = T if and only if the ith square contains j attime t.)

• Hi,t for 0 ≤ i, t ≤ |y|.(Interpretation: Hi,t = T if and only if the reading head is over theith square at time t.)

• Sk,t for 0 ≤ t ≤ |y| and 1 ≤ k ≤ s.(Interpretation: Sk,t = T if and only if M is in state qk at time t.)

The constructed formula with these variables will be of the form

ϕ = A ∧B ∧ C ∧D ∧ E ∧ F ∧G

34

where each conjunct will express the following.

A: At each time t the reading head scans exactly one square.

B: At each time t and for each i the ith square contains exactly one symbol.

C: At each time t, M is in exactly one state.

D: Only the symbol under the reading head can be changed at time t.

E: At each time t, the symbol under the reading head, and the position ofthe reading head and the state for time t + 1 is updated.

F: The start configuration with input x holds at t = 0.

G: The computation contains the accepting state.

It should be clear that an evaluation satisfies ϕ if and only if the corre-sponding interpretation gives an accepting computation.

The formula A is expressed as:∧0≤t≤|y|

((∨

0≤i≤|y|

Hi,t)(∧

0≤i<j≤|y|

(¬Hi,t ∨ ¬Hj,t))).

Formulas B and C are similar and left as an exercise.The formula G is ∨

0≤t≤|y|

Sk,t

where qk is the accepting state.F is the formula

S1,0 ∧H0,0 ∧∧

0≤i<|x|

Ci,ji,0 ∧∧

|x|≤i≤|y|

Ci,0,0

where x = j0j1 . . . j|x|−1.D is the formula ∧

i,j,t

[(Ci,j,t ↔ Ci,j,t+1) ∨Hi,t].

Finally E is expressed as∧i,j,k,t

[(Ci,j,t ∧Hi,t ∧ Sk,t) →∨l

(Ci,jl,t+1 ∧ Skl,t+1 ∧Hil,t+1)],

where l varies over all possible values (q, a, D) according to the δ function.More precisely, if (qr, jr, R) ∈ δ(qi, j) then we get for that value the conclu-sion formula Ci,jr,t+1 ∧ Sr,t+1 ∧Hi+1,t+1, and if (qr, jr, L) ∈ δ(qi, j) then weget Ci,jr,t+1 ∧ Sr,t+1 ∧Hi−1,t+1.

35

Note that D and E are not in conjunctive norml form. To translate theseinto conjunctive normal form is polynomial, in fact it is linear at least if ifwe restrict attention to machines with at most two choices for each δ(q, a).(The latter can always be done without loss of generality.)

Now it only remains to convince oneself that the function which given(M,x, y) produces the formula ϕ is computed in polynomial time. This isleft to the reader.

Having a natural and expressive NP-complete problem such as SATone can now more easily show that many other natural problems are NP-complete or NP-hard by showing that SAT is polynomial time reducible tothem. This is, however, another course.

7 Further reading

Here are some books for further reading on computability theory (recursiontheory) and complexity theory.

Computability theory

• S. B. Cooper, Computability Theory, Chapman Hall / CRC Math-ematics Series, 2003.(An excellent book, a modern presentation of recursion theory or, asthe author prefers, computability theory.)

• H. Rogers, Theory of Recursive Functions and Effective Computabil-ity, McGraw-Hill, 1967.(This is the classic text on recursion theory.)

• R. I. Soare, Recursively Enumerable Sets and Degrees, Springer-Verlag, 1980.(Besides a general introduction the book includes advanced topics inthe theory of r.e. sets and degrees.)

• J. R. Shoenfield, Degrees of Unsolvability, North-Holland, 1971.(A nice little book about the theory of Turing degrees. The presenta-tion is very clear in its informality.)

• P. G. Odifreddi, Classical Recursion Theory volumes I and II, North-Holland, 1989 and 1999.(This is an encyclopedic treatment of recursion theory.)

36

Complexity theory

• J. L. Balcazar, J. Diaz, J.Gabarro, Structural Complexity I,Second Edition, Springer-Verlag, 1995.

• J. L. Balcazar, J. Diaz, J.Gabarro, Structural Complexity II,Springer-Verlag, 1990.

• C. Papadimitriou, Computational Complexity, Addison-Wesley,1994.

References

[1] M. Agrawal, N. Kayal and N. Saxena, PRIMES is in P,manuscript. http://www.cse.iitk.ac.in/news/primality.html

[2] A. Borodin, Computational complexity and the existence of complex-ity gaps, J. Assoc. Comput. Mach. 19(1972), 158 – 174.

[3] A. Cobham, The intrinsic computational difficulty of functions, Pro-ceedings of the 1964 Congress on Logic, Methodology and Philosophy ofScience, North-Holland, 24 – 30, 1964.

[4] S. Cook, The complexity of theorem proving, Proc. 3rd ACM Symp.on Theory of Computing, 1971, 151 – 158.

[5] J. Edmonds, Paths, trees and flowers, Can J. Math 17(1965), 449 –467.

[6] J. Edmonds, “Systems of distinct representatives and linear algebra”,and “Optimum branching”, J. Res. National Bureau of Standards, PartB, 17B (1967), 241 – 245 and 230 – 240.

[7] M. R. Garey and D. S. Johnson, Computers and Intractability. Aguide to the theory of NP-completeness, W. H. Freeman and Com-pany, San Francisco, 1979.

[8] L. G. Khachiyan, A polynomial algorithm in linear programming,Soviet Mathematics Doklady, 20(1979), 191 – 194.

[9] L. A. Levin, Universal sorting problems, Problems of InformationTransmission 9(1973), 265 – 266.

[10] E. L. Post, Recursively enumerable sets of positive integers and theirdecision problems, Bull. Amer. Math. Soc. 50(1944), 284 – 316.

[11] W. J. Savitch, Relationships between nondeterministic and determin-istic tape complexities, J. Comp. System Sci., 4(1970), 177 – 192.

37

[12] A. M. Turing, On computable numbers, with an application to theentscheidungsproblem, Proc. London Math. Soc., Ser. 2, 42(1936), 230– 265.

38