Computational complexity for physicists - Computing in...

MAY/JUNE 2002 31

L I M I T S O FC O M P U T A T I O N

Compared to the traditionally close rela-tionship between physics and mathe-matics, an exchange of ideas and meth-ods between physics and computer

science barely exists. However, the few interac-tions that have gone beyond Fortran program-ming and the quest for faster computers have beensuccessful and have provided surprising insights inboth fields. This is particularly true for the mutualexchange between statistical mechanics and thetheory of computational complexity. Here, I discussthis exchange in a manner directed at physicistswith little or no knowledge of the theory.

The measure of complexityThe branch of theoretical computer science

known as computational complexity is con-cerned with classifying problems according tothe computational resources required to solvethem (for additional information about this field,see the “Related Works” sidebar). What can bemeasured (or computed) is the time that a par-ticular algorithm uses to solve the problem. Thistime, in turn, depends on the algorithm’s imple-

mentation as well as the computer on which theprogram is running.

The theory of computational complexity pro-vides us with a notion of complexity that islargely independent of implementation detailsand the computer at hand. Its precise definitionrequires a considerable formalism, however.This is not surprising because it is related to ahighly nontrivial question that touches the foun-dation of mathematics: What do we mean when wesay a problem is solvable? Thinking about thisquestion leads to Gödel’s incompleteness theo-rem, Turing machines, and the Church-Turingthesis on computable functions.

Here we adopt a more informal, pragmaticviewpoint. A problem is solvable if a computerprogram written in your favorite programminglanguage can solve it. Your program’s runningtime or time complexity must then be defined withsome care to serve as a meaningful measure ofthe problem’s complexity.

Time complexityIn general, running time depends on a prob-

lem’s size and on the specific input data—the in-stance. Sorting 1,000 numbers takes longer thansorting 10 numbers. Some sorting algorithmsrun faster if the input data is partially sorted al-ready. To minimize the dependency on the spe-cific instance, we consider the worst-case timecomplexity T(n):

COMPUTATIONAL COMPLEXITYFOR PHYSICISTS

The theory of computational complexity has some interesting links to physics, in particularto quantum computing and statistical mechanics. This article contains an informalintroduction to this theory and its links to physics.

STEPHAN MERTENS

Otto-von-Guericke University, Magdeburg

1521-9615/02/$17.00 © 2002 IEEE

Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 30, 2009 at 15:47 from IEEE Xplore. Restrictions apply.

32 COMPUTING IN SCIENCE & ENGINEERING

(1)

where t(x) is the algorithm’s running time for in-put data x (in arbitrary units), and the maximumis taken over all problem instances of size n. Theworst-case time is an upper bound for the ob-servable running time.

A measure of time complexity should be basedon a unit of time that is independent of the spe-cific CPU’s clock rate. Such a unit is provided bythe time it takes to perform an elementary op-eration such as adding two integer numbers.Measuring the time in this unit means countingthe number of elementary operations your algo-rithm executes. This number in turn depends

strongly on the algorithm’s implementation de-tails—smart programmers and optimizing com-pilers will try to reduce it. Therefore, rather thanconsider the precise number T(n) of elementaryoperations, we only consider the asymptotic be-havior of T(n) for large values of n as the Lan-dau symbols O and Θ denote:

• We say T(n) is of order at most g(n) and writeT(n) = O (g(n)) if positive constants c and n0 ex-ist such that T(n) ≤ cg(n) for all n ≥ n0.

• We say T(n) is of order g(n) and write T(n)=Θ(g(n)) if positive constants c1, c2, and n0 existsuch that c1g(n) ≤ T(n) ≤ c2g(n) for all n ≥ n0.

Multiplying two n × n matrixes requires n3

multiplications, according to the textbook for-mula. However, this does not mean that theproblem of multiplying two n × n matrices hascomplexity Θ(n3). The textbook formula is a par-ticular algorithm, and an algorithm’s time com-plexity is only an upper bound for a problem’s in-herent complexity. In fact, researchers have foundfaster matrix multiplication algorithms with com-plexity O (nα) and α < 3 during the last decades—the current record being α = 2.376.1 Because theproduct matrix has n2 entries, α cannot be smallerthan 2; it is an open question whether an algo-rithm can achieve this lower bound.

A problem where the upper bound from algo-rithmic complexity meets an inherent lowerbound is sorting n items. Under the general as-sumption that comparisons between pairs ofitems are the only source of information aboutthem, Θ(n log n) is a lower bound for the num-ber of comparisons to sort n items in the worstcase.2 This bound is met by algorithms such as“heapsort” or “mergesort.”

Problem sizeOur measure of time complexity still depends

on the somewhat ambiguous notion of problemsize. In the matrix multiplication example, wetacitly took the number n of rows of one inputmatrix as the “natural” measure of size. Usingthe number of elements m = n2 instead speeds upthe O (n3) algorithm to O (m3/2) without chang-ing a single line of program code. An unam-biguous definition of problem size is required tocompare algorithms.

In computational complexity, all problemssolvable by a polynomial algorithm—that is, analgorithm with time complexity Θ(nk) for somek—are lumped together and called tractable.Problems that can only solvable by algorithms

T n t x

x n( ) max ( ),=

=

Related WorkFor introductions into the field of computational complexity, see

E.L. Lawler et al., eds., The Traveling Salesman Problem, Wiley-Interscience Seriesin Discrete Mathematics and Optimization, John Wiley & Sons, New York, 1985.

H.R. Lewis and C.H. Papadimitriou, “The Efficiency of Algorithms,” ScientificAmerican, vol. 109, no. 1, Jan. 1978, pp. 96–109.

C.H. Papadimitriou and K. Steiglitz, Combinatorial Optimization, Prentice-Hall,Englewood Cliffs, N.J., 1982.

For a deep understanding of the field, read the following classictextbooks:

M.R. Garey and D.S. Johnson, Computers and Intractability: A Guide to the Theoryof NP-Completeness, W.H. Freeman, New York, 1997.

C.H. Papadimitriou, Computational Complexity, Addison-Wesley, Reading,Mass., 1994.

The complexity of Ising spin systems is discussed in:

B. Hayes, “Computing Science: The World in a Spin,” American Scientist, vol. 88, no.5, Sept./Oct. 2000, pp. 2384–388; www.amsci.org/amsci/issues/comsci00/compsci2000-09.html.

For an entertaining introduction (written by a physicist) toGödel’s incompleteness theorem, Turing machines, and theChurch-Turing Thesis on computable functions, read

R. Penrose, The Emperor’s New Mind, Oxford Univ. Press, 1989.

To learn more about quantum parallelism, see www.qubit.org or

D. Aharonov, “Quantum Computation,” Ann. Rev. Computational Physics VI,D. Stauffer, ed., World Scientific, 1998.

Phase transitions in computational complexity are discussed in:

O. Dubois et al., eds., “Phase Transitions in Combinatorial Problems,” specialissue of Theor. Comp. Sci., vol. 265, nos. 1–2, 2001.

B. Hayes, “Computing Science: The Easiest Hard Problem,”American Scientist,vol. 90, no. 2, Mar./Apr. 2002, pp. 113–117; www.amsci.org/amsci/issues/comsci02/compsci2002-03.html.

B. Hayes, “Computing Science: Can’t Get No Satisfaction,” American Scientist,vol. 85, no. 2, Mar./Apr. 1997, pp. 108–112; www.amsci.org/amsci/issues/Comsci97/compsci9703.html.


MAY/JUNE 2002 33

with nonpolynomial running time, such as Θ(2n)or Θ(n!), are also lumped together and called in-tractable. There are practical as well as theoreti-cal reasons for this rather coarse classification.One of the theoretical advantages is that it doesnot distinguish between the O (n3) and O (m3/2)algorithm example from above; hence, we canafford some sloppiness and stick with our am-biguous natural measure of problem size.

Tractable and intractable problemsThe terms tractable and intractable for prob-

lems with polynomial and exponential algo-rithms refer to the fact that an exponential al-gorithm means a hard limit for the accessibleproblem sizes. Suppose that, with your currentequipment, you can solve a problem of size njust within your schedule. If your algorithm hascomplexity Θ(2n), a problem of size n + 1 needstwice the time, pushing you off schedule. Theincrease in time caused by an Θ(n) or Θ(n2) al-gorithm, on the other hand, is far less dramaticand can easily be compensated for by upgrad-ing your hardware.

You might argue that a Θ(n100) algorithm out-performs a Θ(2n) algorithm only for problemsizes that will never occur in your application. Apolynomial algorithm for a problem usually goeshand in hand with a mathematical insight into thatproblem, which lets you find a polynomial algo-rithm with a small degree, typically Θ(nk), k = 1,2, or 3. Polynomial algorithms with k > 10 arerare and arise in rather esoteric problems. It isthis mathematical insight (or rather the lack ofit) that turns intractable problems into a chal-lenge for algorithm designers and computer sci-entists. Let’s look at a few examples of tractableand intractable problems to learn more aboutwhat separates one from the other.

Tractable treesConsider the following problem from network

design. You have a business with several offices,and you want to lease phone lines to connectthem. The phone company charges differentamounts to connect different pairs of cities, andyour task is to select a set of lines that connectsall your offices with a minimum total cost.

In mathematical terms, the cities and the linesbetween them form the vertices V and edges Eof a weighted graph G = (V, E), the weight of anedge being the corresponding phone line’s leas-ing costs. Your task is to find a subgraph thatconnects all vertices in the graph (for example,

a spanning subgraph) and whose edges haveminimum total weight. Your subgraph shouldnot contain cycles because you can always re-move an edge from a cycle, keeping all nodesconnected and reducing the cost. A graphwithout cycles is a tree, so what you are look-ing for is a minimum spanning tree (MST) in aweighted graph (see Figure 1). So, given aweighted graph G = (V, E) with nonnegativeweights, find a spanning tree T ⊆ G with min-imum total weight.

How do you find an MST? A naive approachis to generate all possible trees with n verticesand keep the one with minimal weight. The enu-meration of all trees can be done using Prüfercodes,3 but Cayley’s formula tells us that thereare nn–2 different trees with n vertices. Alreadyfor n = 100 there are more trees than atoms inthe observable universe. Hence, exhaustive enu-meration is prohibitive for all but the smallesttrees. The mathematical insight that turns MSTinto a tractable problem is this:

Lemma: Let U ⊂ V be any subset of the ver-tices of G = (V, E), and let e be the edge with thesmallest weight of all edges connecting U and V –U. Then e is part of the MST.

Proof (by contradiction): Suppose T is an MSTnot containing e. Let e = (u, v), with u ∈ U and v∈ V – U. Then, because T is a spanning tree, itcontains a unique path from u to v that togetherwith e forms a cycle in G. This path must includeanother edge f connecting U and V – U. Now T + e – f is another spanning tree with less totalweight than T. So T was not an MST.

The lemma lets an MST grow edge by edge—using Prim’s algorithm, for example:

Prim(G)Input: weighted graph G(V, E)Output: minimum spanning tree T ⊆ G

10

4

78

4

8

117

2

6

1 2

14

9

Figure 1. A weighted graph and its minimumspanning tree (shaded edges).



beginLet T be a single vertex v from Gwhile T has less than n vertices

find the minimum edge connecting T to G – T

add it to Tend

end

The precise time complexity of Prim’s algorithmdepends on the data structure used to organize theedges, but in any case, O(n2log n) is an upper bound(faster algorithms are discussed elsewhere3).Equipped with such a polynomial algorithm, youcan find MSTs with thousands of nodes within sec-onds on a personal computer. Compare this to anexhaustive search. According to our definition,finding an MST is a tractable problem.

Intractable itinerariesEncouraged by the efficient algorithm for an

MST, let us now investigate a similar problem.Your task is to plan an itinerary for a travelingsalesman who must visit n cities. You are given amap with all cities and the distances betweenthem. In what order should the salesman visitthe cities to minimize the total distance traveled?You number the cities arbitrarily and write downthe distance matrix (dij), where dij denotes thedistance between city i and city j. A tour is givenby a cyclic permutation π: [1...n]→[1...n], whereπ(i) denotes the successor of city i. Your prob-lem can be defined as

The Traveling Salesman Problem (TSP): Givenan n × n distance matrix with elements dij ≥ 0,find a cyclic permutation π : [1...n] → [1...n] thatminimizes

(2)

The TSP is probably the mostfamous optimization problem (see www.keck.caam.rice.edu/tsp).Finding good solutions, even tolarge problems, is not difficult, buthow can we find the best solutionfor a given instance? There are (n– 1)! cyclic permutations, and cal-culating the length of a single tourtakes O (n). So, an exhaustivesearch has complexity O (n!).Again this approach is limited tovery small instances.

Is there a mathematical insight that providesa shortcut to the optimum solution, such as foran MST? Nobody knows. Despite many efforts,researchers have not found a polynomial algo-rithm for the TSP. There are some smart and ef-ficient (that is, polynomial) algorithms that findgood solutions but do not guarantee yielding theoptimum solution.4 According to this definition,the TSP is intractable.

Why is the TSP intractable? Again, nobodyknows. There is no proof that excludes the exis-tence of a polynomial algorithm for the TSP, somaybe someday somebody will come up with apolynomial algorithm and the correspondingmathematical insight. This is very unlikely, how-ever, as we will see soon.

The TSP’s intractability astonishes all themore considering the tractability of a very simi-lar, almost identical problem, the Assignmentproblem:

Assignment: Given an n × n cost matrix with el-ements dij ≥ 0, find a permutation π : [1...n] →[1...n] that minimizes

(3)

The only difference between the TSP and As-signment is that the latter allows all permuta-tions on n items, not just the cyclic ones. If dijdenotes distances between cities, Assignmentcorresponds to total tour length minimizationfor a variable number of salesmen, each travel-ing his own subtour (see Figure 2).

The classical application of Assignment is theassignment of n tasks to n workers, subject to theconstraint that each worker is assigned exactly

c dn i i

i

n

( ) .( )π π==

∑1

c dn i i

i

n

( ) .( )π π==

∑1

(b)(a)

Figure 2. Same instance, different problems: A valid configuration of (a) the TravelingSalesman Problem and (b) the Assignment Problem. Whereas the Assignment Problem can be solved in polynomial time, the TSP is intractable.


MAY/JUNE 2002 35

one task. Let dij denote the cost of worker I per-forming task j, and π(i) denote the task assignedto worker i. Assignment is the problem of mini-mizing the total cost.

There are n! possible assignments of n tasks ton workers—hence, exhaustive enumeration isagain prohibitive. In contrast to the TSP, how-ever, we can solve Assignment in polynomialtime—for example, using the O(n3) Hungarianalgorithm.5 Compared to an MST, the algorithmand the underlying mathematical insight are abit more involved and not discussed here.

Decision problemsSo far, we have discussed optimization prob-

lems: solving MST, TSP, or Assignment impliesthat we compare an exponential number of feasi-ble solutions with each other and pick the opti-mum solution. Exhaustive search does this explicitly; polynomial shortcuts implicitly. By in-vestigating simpler problems whose solutions arerecognizable without explicit or implicit compar-ison to all feasible solutions, we might learn moreabout the barrier that separates tractable from in-tractable problems. So, let’s consider decision prob-lems, whose solution is either “yes” or “no.”

We can turn any optimization problem into adecision problem by adding a bound B to the in-stance. For example,

MST (decision): Given a weighted graph G =(V, E) with nonnegative weights and a numberB ≥ 0, does G contain a spanning tree T with to-tal weight ≤ B?

TSP (decision): Given an n × n distance matrixwith elements dij ≥ 0 and a number B ≥ 0, is therea tour π with length ≤ B?

In a decision problem, the feasible solutions arenot evaluated relative to each other but to an ab-solute criterion: a tour in the TSP either haslength ≤ B or it doesn’t.

We can solve MST(D) in polynomial time bysimply solving the optimization variant MSTand comparing the result to the parameter B.For the TSP(D), this approach does not help. Infact, we see later that a polynomial algorithm ex-ists for the TSP(D) if and only if there exists apolynomial algorithm for the TSP. It seems as ifwe cannot learn more about the gap betweentractable and intractable problems by consider-ing decision variants of optimization problems.So let’s look at other decision problems not de-rived from optimization problems.

Eulerian circuitsOur first genuine decision problem dates back

to the 18th century, where in the city of Königs-berg (now Kaliningrad), seven bridges crossedthe river Pregel and its two arms (see Figure 3).A popular puzzle of the time asked if it were pos-sible to walk through the city crossing each ofthe bridges exactly once and returning home.

Leonhard Euler solved this puzzle in 1736.6First, he recognized that to solve the problem,the only thing that matters is the pattern of in-terconnections of the banks and islands—a graphG = (V, E) in modern terms. The graph corre-sponding to the puzzle of the Königsbergbridges has four vertices for the two banks andtwo islands and seven edges for the bridges (seeFigure 3). Euler’s paper on the Königsbergbridges is the birth of graph theory.

To generalize the Königsberg bridges prob-lem, we need some terminology from graph the-ory.3 A walk in a graph G = (V, E) is an alternat-ing sequence of vertices v ∈ V and edges (v, v′) ∈E: v1, (v1, v2), v2, (v2, v3), ..., (vl–1, vl), vl. Note thatthe sequence begins and ends with a vertex, andeach edge is incident with the vertices immedi-ately preceding and succeeding it. A walk istermed closed if vl = v1; otherwise, it is open. Awalk is called a trail if all its edges are distinct,and a closed trail is called a circuit. What thestrollers in Königsberg tried to find was a circuit

fb

B

D

C

A

cd

e

g

a

Figure 3. The seven bridgesof Königsberg, (a) as drawnin Leonhard Euler’s paperfrom 17366 and (b)represented as a graph. Inthe graph, the riverbanksand islands are condensed topoints (vertices), and each ofthe bridges is drawn as a line(edge).

c C

c

A

bB

D

e

d

f

(a) (b)



that contains all edges. In honor of Euler, such acircuit is called an Eulerian circuit.

We can now define the generalization of theKönigsberg bridges problem:

Eulerian Circuit: Given a graph G = (V, E),does G contain an Eulerian circuit?

Obviously, this is a decision problem. The an-swer is either “yes” or “no,” and we can check tosee whether a circuit is Eulerian without resort-ing to all possible circuits.

Once again, an exhaustive search would solvethis problem, but Euler noticed the intractabilityof this approach. More than 200 years before theadvent of computers, he wrote,

The particular problem of the seven bridges ofKönigsberg could be solved by carefully tabulat-ing all possible paths, thereby ascertaining by in-spection which of them, if any, met the require-ment. This method of solution, however, is tootedious and too difficult because of the largenumber of possible combinations, and in otherproblems where many more bridges are involvedit could not be used at all.7

He solved the Königsberg bridges problem notby listing all possible trails but by using mathe-matical insight. He noticed that in a circuit, youmust leave each vertex via an edge different fromthe edge that has taken you there. In otherwords, the vertex’s degrees (that is, the numberof edges adjacent to it) must be even. This is ob-viously a necessary condition, but Euler provedthat it is also sufficient:

Theorem: A connected graph G = (V, E) con-tains an Eulerian circuit if and only if the degreeof every vertex v ∈ V is even.

Euler’s theorem lets us devise a polynomial al-gorithm for the Eulerian Circuit: Check the de-gree of every vertex in the graph. If one vertexhas an odd degree, return “no.” If all verticeshave an even degree, return “yes.” This algo-rithm’s running time depends on the graph’s en-coding. If G = (V, E) is encoded as a |V| × |V|adjacency matrix with entries aij = number ofedges connecting vi and vj, the running time isO(|V|2).

Thanks to Euler, the Eulerian Circuit is atractable problem. The burghers of Königsberg,on the other hand, had to learn from Euler thatthey would never find a walk through theirhometown crossing each of the seven bridges ex-actly once.

Hamiltonian cyclesAnother decision problem is associated with

the mathematician and Astronomer Royal of Ire-land, Sir William Rowan Hamilton. In 1859,Hamilton put on the market a new puzzle calledthe Icosian game (see Figure 4).

Generalizing the Icosian game calls for somemore definitions from graph theory: A closedwalk in a graph is called a cycle if all its vertices (ex-cept the first and the last) are distinct. A Hamil-tonian cycle is one that contains all vertices of agraph. The generalization of the Icosian gamethen reads

Hamiltonian Cycle: Given a graph G = (V, E),does G contain a Hamiltonian cycle?

There is a certain similarity between the Euler-ian Circuit and Hamiltonian Cycle. In the for-mer, we must pass each edge once—in the lat-ter, each vertex once. Despite this resemblance,the two problems represent entirely different de-grees of difficulty. The available mathematicalinsights into the Hamiltonian Cycle provide usneither with a polynomial algorithm nor with aproof that such an algorithm is impossible. TheHamiltonian Cycle is intractable, and nobodyknows why.

ColoringImagine we wish to arrange talks in a congress

in such a way that no participant will be forcedto miss a talk he or she wants to hear. Assuming agood supply of lecture rooms that lets us hold as

(b)(a)

Figure 4. Sir Hamilton’s Icosian game: (a) Find a route along theedges of the dodecahedron, passing each corner exactly once andreturning to the starting corner. (b) A solution is indicated (shadededges) in the planar graph that is isomorphic to the dodecahedron.


MAY/JUNE 2002 37

many parallel talks as we like, can we finish theprogram within k time slots? We can formulatethis problem in terms of graphs: Let G be a graphwhose vertices are the talks and in which twotalks are adjacent (joined by an edge) if and onlyif there is a participant wishing to attend both.Your task is to assign one of the k time slots toeach vertex in such a way that adjacent verticeshave different slots. The common formulation ofthis problem uses colors instead of time slots:

k-Coloring: Given a graph G = (V, E), is there acoloring of the vertices of G using at most k dif-ferent colors such that no two adjacent verticeshave the same color?

When k = 2, this problem is tractable—construct-ing a polynomial algorithm is an easy exercise. Fork = 3, however, things change considerably: 3-Col-oring is intractable. Note that for larger k, theproblem gets easier again: a planar graph is alwayscolorable with four colors. This is the famous 4-Color Theorem. 3-Coloring remains intractableeven when restricted to planar graphs.

SatisfiabilityI close this section with a decision problem

that is not from graph theory but from Booleanlogic. A Boolean variable x can take on the value0 (false) or 1 (true). Boolean variables can becombined in clauses using the Boolean operators

• NOT (negation): the clause is true( ) if and only if x is false (x = 0).

• AND ∧ (conjunction): the clause x1 ∧ x2 istrue (x1 ∧ x2 = 1) if and only if both variablesare true: x1 = 1 and x2 = 1.

• OR ∨ (disjunction): the clause x1 ∨ x2 is true(x1 ∨ x2 = 1) if and only if at least one of thevariables is true: x1 = 1 or x2 = 1.

A variable x or its negation is called a literal.Different clauses can combine to yield complexBoolean formulas such as

. (4)

A Boolean formula evaluates to either 1 or 0, de-pending on the assignment of the Boolean vari-ables. For example, F1 = 1 for x1 = 1, x2 = 1, x3 =0, and F1 = 0 for x1 = x2 = x3 = 1. A formula F iscalled satisfiable if there is at least one assignmentof the variables such that the formula is true. F1is satisfiable,

(5)

is not.We can write every Boolean formula in con-

junctive normal form (CNF)—that is, as a set ofclauses Ck combined exclusively with the ANDoperator

F = C1 ∧ C2 ∧ ... ∧ Cm, (6)

where the literals in each clause are combinedexclusively with the OR operator. The examplesF1 and F2 are both written in CNF. Each clausecan be considered a constraint on the variables,and satisfying a formula means satisfying a set of(possibly conflicting) constraints simultaneously.Hence, consider the following as a prototype ofa constraint satisfaction problem:8

Satisfiability (SAT): Given disjunctive clausesC1, C2, ..., Cm of literals, where a literal is a vari-able or negated variable from the set {x1, x2, ...,xn}, is there an assignment of variables that satis-fies all clauses simultaneously?

Fixing the number of literals in each clause leads to

k-SAT: Given disjunctive clauses C1, C2, ..., Cmof k literals each, where a literal is a variable ornegated variable from the set {x1, x2, ..., xn}, isthere an assignment of variables that satisfies allclauses simultaneously?

Polynomial algorithms are known for 1-SATand 2-SAT.9 No polynomial algorithm is knownfor general SAT and k-SAT if k > 2.

Complexity classesNow we have seen enough examples to intro-

duce two important complexity classes for deci-sion problems and to discuss how these classesrelate to other kinds of problems.

Tractable problems Defining the class of tractable decision problems

is easy: it consists of those problems for which apolynomial algorithm is known. The corre-sponding class is named P for “polynomial”:

Definition: A decision problem P is element ofthe class P if and only if a polynomial time algo-rithm can solve it.

Eulerian Circuit, 2-Coloring, MST(D), and soforth are in P .

F x x x x x x2 1 2 1 2 2 1( , ) ( )= ∨ ∧ ∧

F x x x x x xx x x x x x1 1 2 3 1 2 3

2 3 1 2 1 3

( , , ) ( )( ) ( ) ( )

= ∨ ∨ ∧∨ ∧ ∨ ∧ ∨

x

x = 1 x⋅



Nondeterministic algorithmsThe definition of the second complexity class

involves the concept of a nondeterministic algo-rithm, which is like an ordinary algorithm, ex-cept that it might use one additional, very pow-erful instruction:10

goto both label 1, label 2

This instruction splits the computation into twoparallel processes, one continuing from each ofthe instructions, indicated by “label 1” and “label2.” By encountering more such instructions, thecomputation branches like a tree into severalparallel computations that potentially can growas an exponential function of the time elapsed(see Figure 5). A nondeterministic algorithm canperform an exponential number of computationsin polynomial time.

In the world of conventional computers, non-deterministic algorithms are a theoretical con-cept only, but this could change in quantumcomputing. We need the concept of nondeter-minism to define the class NP of “nondeter-ministic polynomial” problems:

Definition: A decision problem P is in the classNP if and only if a nondeterministic algorithmcan solve it in polynomial time.

Solubility by a nondeterministic algorithm meansthis: All branches of the computation will stop, re-turning either “yes” or “no.” We say that the over-all algorithm returns “yes” if any of its branches

returns “yes.” The answer is “no” if none of thebranches reports “yes.” We say that a nondeter-ministic algorithm solves a decision problem inpolynomial time if the number of steps used by thefirst of the branches to report “yes” is bounded bya polynomial in the problem’s size.

We require polynomial solubility only for a de-cision problem’s “yes” instances. This asymmetrybetween “yes” and “no” reflects the asymmetrybetween the “there is” and “for all” quantifiers indecision problems. A graph G is a “yes” instanceof the Hamiltonian Cycle if there is at least oneHamiltonian cycle in G. For a “no” instance, allcycles in G must be non-Hamiltonian.

Conventional (deterministic) algorithms arespecial cases of nondeterministic algorithms(those nondeterministic algorithms that do notuse the goto both instruction). It follows imme-diately that P ⊆ NP.

All decision problems discussed thus far havebeen members of NP. Here’s a nondeterminis-tic polynomial algorithm for SAT:

Satisfiability (F)Input: Boolean formula F(x1, ..., xn)Output: ‘yes’ if F is satisfiable, ‘no’ otherwisebegin

for i = 1 to ngoto both label 1, label 2

label 1: xi = true; continuelabel 2: xi = false; continue

endif F(x1, ..., xn) = true then return ‘yes’

else return ‘no’end

The for loop branches at each iteration: in onebranch, xi = true; in the other branch, xi = false(the continue instruction starts the loop’s nextiteration). After executing the for loop, we have2n branches of computation—one for each of thepossible assignments of n Boolean variables.

The power of nondeterministic algorithms isthat they allow the exhaustive enumeration of anexponentially large number of candidate solu-tions in polynomial time. If the evaluation ofeach candidate solution (calculating F(x1, ..., xn)in the SAT example) in turn can be done in poly-nomial time, the total nondeterministic algo-rithm is polynomial. For a problem from theclass NP, the sole source of intractability is theexponential size of the search space.

Succinct certificatesThere is a second, equivalent definition of

no

no no

yes

no

no yes

no

no

no

no nono

Elapsedtime

Figure 5. Example of a nondeterministic algorithm’s execution history.


MAY/JUNE 2002 39

NP based on the notion of a succinct certificate.A certificate is a proof. If you claim that a graphG has a Hamiltonian Cycle, you can proof yourclaim by providing a Hamiltonian Cycle. Cer-tificates for the Eulerian Circuit and k-Coloringare an Eulerian Circuit and a valid coloring, re-spectively. A certificate is succinct if its size isbounded by a polynomial in the problem size.The second definition then reads

Definition: A decision problem P is element ofthe class NP if and only if for every “yes” in-stance of P there exists a succinct certificate thatcan be verified in polynomial time.

The equivalence between both definitions canbe proven easily.10 The idea is that a succinctcertificate can deterministically select the branchin a nondeterministic algorithm that leads to a“yes” output.

The definition based on nondeterministic al-gorithms reveals the key feature of the class NPmore clearly, but the second definition is moreuseful for proving that a decision problem is inNP. For example, consider

Compositeness: Given a positive integer N, arethere integer numbers p > 1 and q > 1 such that N= pq?

A certificate of a “yes” instance N of Compos-iteness is a factorization N = pq. It is succinct be-cause the number of bits in p and q is less thanor equal to the number of bits in N, and it can

be verified in quadratic time by multiplication.So, Compositeness ∈ NP.

Most decision problems ask for the existence ofan object with a given property, such as a cycle,which is Hamiltonian or a factorization with inte-ger factors. In these cases, the desired object mightserve as a succinct certificate. For some problems,this does not work, however. For example,

Primality: Given a positive integer N, is N prime?

Primality is the negation or complement ofCompositeness: the “yes” instances of the for-mer are the “no” instances of the latter and viceversa. A succinct certificate for Primality is byno means obvious. In fact, for many decisionproblems in NP, no succinct certificate isknown for the complement—that is, whether thecomplement is also in NP is not known. ForPrimality, however, there is a succinct certificatebased on Fermat’s Theorem.11 Hence, Primal-ity ∈ NP .

A first map of complexityFigure 6a summarizes what we have achieved

so far. The class NP consists of all decisionproblems whose sole source of difficulty is thesize of the search space, which grows exponen-tially with the problem size. These problems areintractable unless some mathematical insightprovides us with a polynomial shortcut to avoidan exhaustive search. Such an insight promotes aproblem into the class P of polynomially solu-ble problems.

(b)(a)

Compositeness

2-ColoringEulerian Circuit

Assignment(D)MST(D)

2-SAT

3-Coloring

Graph-Isomorphism

SAT

3-SAT

Hamiltonian Cycle TSP(D)

Primality

P

NP

2-ColoringEulerian Circuit

Assignment(D)MST(D)

2-SAT

Graph-Isomorphism

SAT

TSP(D)

CompositenessPrimality

Hamiltonian Cycle

3-Coloring3-SAT

PP P

NP

NP complete

Figure 6. (a) A first map of complexity; (b) the map of complexity revisited. All problems indicated are defined within thetext. Problems with a (D) are decision variants of optimization problems.



The class NP not only contains many prob-lems with important applications but also repre-sents a real challenge: all problems in NP stillhave a chance to be in P. A proof of nonexis-tence of a polynomial algorithm for a singleproblem from NP would establish that P ≠NP. As long as such a proof is missing,

(7)

represents the most famous open conjecture intheoretical computer science.

NP completenessSo far, all the intractable problems seem to be

isolated islands in the map of complexity. In fact,they are tightly connected by a device called poly-nomal reduction, which lets us map the computa-tional complexity of one problem to the com-plexity of another. This finally leads to thesurprising result that there are some intractableproblems that are equivalent to all other prob-lems. Each of these so called NP-completeproblems embodies the secret of intractability,since a polynomial time algorithm for one ofthem would immediately imply the tractabilityof all problems in NP.

Polynomial reductions. The computational com-plexities of two problems, P1 and P2, can be re-lated to each other by constructing an algorithmfor P1 that uses an algorithm for P2 as a “sub-routine.” Consider the following algorithm thatrelates Hamiltonian Cycle to TSP(D):

Hamiltonian Cycle (G)Input: Graph G = (V, E)Output: ‘yes’ if G contains a Hamiltonian cycle, ‘no’ otherwise(1) begin(2) n : = |V|(3) for i = 1 to n(4) for j = 1 to n(5) if (vi, vj) ∈ E then dij := 1(6) else dij := 2(7) if TSP-decision (d, B := n) = ‘yes’

then return ‘yes’(8) else return ‘no’(9) end

This algorithm solves the Hamiltonian Cycleby solving an appropriate instance of theTSP(D). In the for loops (lines 3 through 5), adistance matrix d is set up with entries dij = 1 ifthere is an edge (vi, vj) in G—otherwise, dij = 2. A

Hamiltonian Cycle in G is a valid tour in thecorresponding TSP with all intercity distanceshaving length 1—that is, with total length n.Conversely, suppose that the TSP has a tour oflength n. Because the intercity distances are ei-ther 1 or 2, and a tour sums up n such distances,a total length n implies that each pair of succes-sively visited cities must have distance 1—thatis, the tour follows existing edges in G and cor-responds to a Hamiltonian Cycle. So, the call toa subroutine that solves TSP(D) (line 7) yields asolution to the Hamiltonian Cycle.

How does this algorithm relate the computa-tional complexity of the Hamiltonian Cycle tothat of the TSP(D)? This is a polynomial algo-rithm if the call to the TSP(D) solver is consid-ered an elementary operation. If someone comesup with a polynomial algorithm for TSP(D), wewill instantly have a polynomial algorithm forthe Hamiltonian Cycle. We say that the Hamil-tonian Cycle is polynomially reducible to TSP(D)and write

Hamiltonian Cycle ≤ TSP(D). (8)

In many books, polynomial reducibility is de-noted by ∝ instead of ≤. We use ≤ because thisnotation stresses an important consequence ofpolynomial reducibility:12 the existence of apolynomial reduction from P1 to P2 excludes thepossibility that we can solve P2 in polynomialtime but not P1. Hence, we can read P1 ≤ P2 asP1 cannot be harder than P2. Here is the (informal)definition:

Definition: We say a problem P1 is polynomi-ally reducible to a problem P2 and write P1 ≤ P2if a polynomial algorithm exists for P1 providedthere is a polynomial algorithm for P2.

NP-complete problems. Here are some other poly-nomial reductions that we can verify, similar toEquation 8 (the corresponding reduction algo-rithms appear elsewhere13):

SAT ≤ 3-SAT3-SAT ≤ 3-Coloring3-Coloring ≤ Hamiltonian Cycle

(9)

Polynomial reducibility is transitive: P1 ≤ P2 andP2 ≤ P3 imply P1 ≤ P3. From transitivity and Equa-tions 8 and 9, it follows that each SAT, 3-SAT, 3-Coloring, and Hamiltonian Cycle reduces toTSP(D)—that is, a polynomial algorithm forTSP(D) implies a polynomial algorithm for allthese problems. This is amazing, but it’s only the

P NP=?


MAY/JUNE 2002 41

beginning. Stephen Cook14 re-vealed polynomial reducibility’strue scope in 1971 when heproved that

Theorem: All problems inNP are polynomially reducibleto SAT,

∀ P ∈ NP: P ≤ SAT. (10)

This theorem means that

• No problem in NP is harderthan SAT, or SAT is amongthe hardest problems in NP.

• A polynomial algorithm forSAT would imply a polyno-mial algorithm for every problem in NP—that is, it would imply P = NP .

It seems as if SAT is very special, but accordingto transitivity and Equations 8 and 9, 3-SAT, 3-Coloring, the Hamiltonian Cycle, or theTSP(D) can replace it. These problems form anew complexity class:

Definition: A problem P is called NP completeif P ∈ NP and Q ≤ P for all Q ∈ NP.

The class of NP-complete problems collects thehardest problems in NP. If any of them has anefficient algorithm, then every problem in NPcan be solved efficiently—thus, P = NP. This isextremely unlikely, however, considered the fu-tile efforts of many brilliant people to find poly-nomial algorithms for problems such as theHamiltonian Cycle or TSP(D).

The map of NP. Since Cook developed his the-orem, many problems have been shown to beNP complete (see www.nada.kth.se/~viggo/wwwcompendium for a comprehensive, up-to-date list of hundreds of NP-complete prob-lems). Thus, the map of NP presented in Fig-ure 6a needs some redesign (see Figure 6b). Itturns out that all the intractable problems wehave discussed so far are NP complete—exceptCompositeness and Primality. For both prob-lems, neither a polynomial algorithm is knownnor a polynomial reduction that classifies themas NP complete. Another NP problem that re-sists classification in either P or NP is

Graph Isomorphism: Given two graphs G = (V,E) and G´(V, E´) on the same set of nodes, are G

and G´ isomorphic—that is, is there a permuta-tion π of V such that G´ = π (G), where π(G) de-notes the graph (V, {[π(u), π(v)] : [u, v] ∈ E})?

There are more problems in NP that resistclassification in P or NP, but none of theseproblems has been proven not to belong to P orNP. What has been proven is

Theorem: If P ≠ NP, then NP problems existthat are neither in P nor NP complete.

This theorem15 rules out one of three tentativemaps of NP (see Figure 7).

Beyond NPThe notions NP and NP complete strictly

apply only to decision problems (“Is there a so-lution?”). The ideas of this approach can be gen-eralized to optimization problems (“What is thebest solution?”) and counting problems (“Howmany solutions are there?”).

Optimization problems. How does the classifica-tion of decision problems relate to optimizationproblems? The general instance of an optimiza-tion problem is a pair (F, c), where F is the set offeasible solutions and c is a cost function c: F →!. Here I consider only combinatorial optimiza-tion where the set F is countable. A combinator-ial optimization problem P comes in three dif-ferent flavors:

1.The optimization problem P(O): Find thefeasible solution f *∈ F that minimizes the costfunction.

2.The evaluation problem P(E): Find the costc* = c (f *) of the minimum solution.

(c)(b)(a)

NP complete

P

NP complete

P

NP P = NP

PP

P = NP

Figure 7. Three tentative maps of NP. We can rule out map B; map A is likely (but notsurely) the correct map. The discovery of a polynomial algorithm for any of the NP-complete problems would turn C into the correct map.



3.The decision problem P(D): Given a boundB ∈ !, is there a feasible solution f ∈ F suchthat c( f ) ≤ B?

Under the assumption that we can evaluate thecost function c in polynomial time, it is straight-forward to write down polynomial reductionsthat establish

P(D) ≤ P(E) ≤ P(O). (11)

If an optimization problem’s decision variant isNP complete, there is no efficient algorithm forthe optimization problem at all—unless P =NP. An optimization problem such as the TSP,whose decision variant is NP complete, is de-noted NP hard.

Does a polynomial algorithm for a decisionproblem imply a polynomial algorithm for theoptimization or evaluation variant? For that, weneed to prove the reversal of Equation 11:

P(O) ≤ P(E) ≤ P(D). (12)

P(E) ≤ P(D) can be shown to hold if the opti-mum solution’s cost is an integer with logarithmbounded by a polynomial in the input’s size.The corresponding polynomial reduction eval-uates the optimal cost c* by asking, “Is c* ≤ B?”for a sequence of values B that approaches c*(similar to the bisection method for finding afunction’s zeroes).

There is no general method to prove P(O) ≤P(E), but a strategy that often works for the TSP isthis: Let c* be the known solution of TSP(E). Re-place an arbitrary entry dij of the distance matrixwith a value c > c* and solve TSP(E) with this mod-ified distance matrix. If this modification doesn’taffect the tour’s optimum length, the link ij doesnot belong to the optimal tour. Repeating this pro-cedure for different links, we can reconstruct theoptimum tour with a polynomial number of callsto a TSP(E) solver; hence, TSP(O) ≤ TSP(E).

Counting problems. So far, we have studied tworelated styles of problems: Decision problemsask whether a desired solution exists, and opti-mization problems require that a solution beproduced. A third important and fundamentallydifferent kind of problem asks how many solu-tions exist. The counting variant of SAT reads

#SAT: Given a Boolean expression, computethe number of different truth assignments thatsatisfy it.

Similarly, #Hamiltonian Cycle asks for the num-

ber of different Hamiltonian Cycles in a givengraph, #TSP for the number of different tourswith length ≤ B, and so on.

Definition: A counting problem #P is a pair (F,d), where F is the set of all feasible solutions, andd is a decision function d: F → {0,1}. The outputof #P is the number of f ∈ F with d(f) = 1. Theclass #P (pronounced “number P”) consists ofall counting problems associated with a decisionfunction d that can be evaluated in polynomialtime.

Like the class NP, #P collects all problemswhose sole source of intractability is the numberof feasible solutions. A polynomial algorithm fora counting problem #P implies a polynomial al-gorithm for the associated decision problem P:P ≤ #P. Hence, it is unlikely that #SAT can besolved efficiently. In fact, we can define polyno-mial reducibility for counting problems andprove that all problems in #P reduce polynomi-ally to #SAT:6

Theorem: #SAT is #P complete.

As you might have guessed, #Hamiltonian Cy-cle and #TSP are also #P complete. Despite thesimilarity between NP and #P, counting prob-lems are inherently harder than decision prob-lems. This is documented by those #P-completeproblems for which the corresponding decisionproblem can be solved in polynomial time—theclassical example being the problem of calculat-ing a matrix’s permanent.16

Computational complexity and physicsThe relationship between computation com-

plexity and physics should offer new insights,some of which are discussed here. For example,some knowledge of computational complexityhelps us understand the promises and limitationsof quantum computers. In addition, we canseemingly transfer the notion of tractable andintractable problems to the problem of analyticalsolubility of models from statistical physics, ex-plaining to some extent why, for example, theIsing model is exactly soluble in 2D but not 3D.

Another interesting link between computa-tion complexity and physics is that statisticalmechanics offer means for the general proba-bilistic analysis of computational problems. Instatistical mechanics, we typically formulate anoptimization problem as a spin glass and ana-


MAY/JUNE 2002 43

lyze the latter’s low temperature properties.This “physical” approach often yields resultsthat go beyond the results obtained by tradi-tional methods. Another surprising observationis the fact that random instances of intractableproblems can be solved in polynomial time. Toobserve the exponential running time, the pa-rameters of the random ensemble must be ad-justed carefully. This scenario corresponds to aphase transition in physical systems and is there-fore best studied within the framework of sta-tistical mechanics.

Quantum parallelismThere is some theoretical evidence that com-

puters using quantum systems as computationaldevices are more powerful than computers basedon classical devices. The hope is that problemsthat are intractable on a classical computer be-come tractable when put on a quantum com-puter. Results such as Peter Shor’s celebratedquantum algorithm for factorization nurture thishope, but a real breakthrough is still missing.

In a seminal paper, Richard Feynman pointedout that a system of n quantum particles is expo-nentially hard to simulate on a classical com-puter.17 The idea of quantum computing is to re-verse this situation and simulate a classically hard(for example, exponential) problem in polynomialtime on a computer made of quantum devices.

A quantum computer processes qubits—quan-tum two-state systems |0⟩, |1⟩. A quantum com-puter’s key feature is that its registers can hold andprocess linear superpositions of all 2n productstates of n qubits, such as

. (13)

Using this feature, constructing a quantum com-puter capable of computing any function f(x1, ...,xn) of n Boolean variables simultaneously for all2n possible input values is not difficult—in the-ory, at least. This quantum parallelism resemblesa nondeterministic algorithm with its goto bothinstruction and its exponentially branching exe-cution tree. Is quantum parallelism the key toexponential computing power? The problem ishow to extract the exponential information outof a quantum computer. When we defined non-deterministic solubility, we didn’t care about howto spot the single “yes” among the 2n “no” an-swers. This works fine for a theoretical concept,but for a practical computer, reading the outputreally matters.

To gain the advantage of exponential paral-lelism, we must combine it with another quantumfeature known as interference. The goal is toarrange the computation such that constructiveinterference amplifies desired result and destruc-tive interference cancels the rest. Because of theimportance of interference phenomena, it is notsurprising that calculating the Fourier transformwas the first problem that underwent an exponen-tial speedup: from O(n log n) on a classical toO(log2n) on a quantum computer. This speedupwas the seed for the most important quantum al-gorithm known today: Shor’s algorithm to factoran integer in polynomial time.18

Although Shor’s algorithm has some conse-quences for public key cryptography, it does notshatter the world of NP: remember that Com-positeness is in NP, but it is not NP complete.Hence, the holy grail of quantum computing hasyet to be discovered—a polynomial time quan-tum algorithm for an NP-complete problem.

Analytical solubility of Ising modelsSome problems in statistical physics have been

exactly solved, but the majority of problems havewithstood the efforts of generations of mathe-maticians and physicists. Why are some prob-lems analytically solvable, whereas other, oftensimilar, problems are not? Relating this questionto the algorithmic complexity of evaluating thepartition function gives us no final answer buthelps clarify the borderline that separates ana-lytically tractable from intractable problems.

For example, consider the Ising spin glass on ageneral graph G.19 Let σ = (σ1, ..., σN) be an as-signment of Ising spins σi = ±1 (up or down).The energy of a configuration σ is

, (14)

where H is the external magnetic field, Ji,j are thecoupling constants, and the first summation isover all edges in G. The fundamental problemin statistical mechanics is to determine the par-tition function

. (15)

Evaluating the sum directly requires O(2N) op-erations. The notion of analytic solution has noprecise definition, but as a minimum require-ment, we want to reduce this number from beingexponential to being polynomial in N.

Consider the well-known transfer matrix so-

Z G eN

E( ) ( )= −∑ β σ

σ

E J Hi j i j i

ii j

( ) ,,

σ σ σ σ= − − ∑∑

1

21 2

0

1

1 2n n

i i ii i i

n

| ..., ,..., =∑



lution of the 1D Ising glass with periodic bound-ary conditions and coupling Jk between spins σkand σk+1 (see Figure 8):

, (16)

which can be evaluated in O(N) elementary op-erations. Because any exact solution must in-clude all numbers Jk, this is the best we can ex-pect. In the homogeneous case Jk ≡ J, where wecan calculate the product of transfer matrices,

(17)the evaluation complexity drops to O(log N) us-ing fast exponentiation.

Writing the partition sum as

(18)

where the sum is over all possible energy val-ues, it becomes obvious that calculating Z isclosely related to the #P problem of determin-ing n(Ek). For general finite graphs G, thisproblem has proven to be #P complete,20 sothere is no hope of finding an analytical solu-tion (even in the weak sense above). This situ-ation hardly improves if we restrict the graphsto the more “physical” crystal lattices: com-puting the partition function for a finite sub-lattice of a nonplanar crystal lattice is #P com-plete.21 This includes every crystal lattice in d >2, the d = 2 model with next-nearest neighborinteractions, two coupled d = 2 models, and soforth. It also includes all models with d ≥ 2 andexternal field H, because these can be trans-formed into zero-field models on an aug-mented graph (which is nonplanar unlessthe underlying lattice graph G is 1D). Con-

structing from G is easy—just adjoin an ad-ditional vertex (spin) σ0 to G and let the addi-tional edges σ0σi have constant interaction en-ergy J0,i = H. The partition function of theaugmented system reads

,

(19)where the additional factor comes from the newspin σ0 = ±1. From this expression, it is easy tosee that Z( ) equals two times the partitionfunction Z(G) in field H.

But where are the soluble Ising models? It hasbeen proven that we can evaluate the partitionfunction of Ising systems on planar crystal latticesin polynomial time.22 This includes the cele-brated Onsager solution of the square ferro-magnet23 as well as the 1D example just pre-sented. It turns out that we can calculate theIsing model’s partition sum in polynomial timefor all graphs of fixed genus g.24,25 A graph hasgenus g if it can be drawn on an orientable sur-face of genus g (a sphere with g “handles” at-tached to it) without crossing the edges. Planargraphs have genus 0, toroidal graphs have genus1, and so on. For the crystal lattices in d > 2, thegenus increases monotonically with the numberof spins—that is, it is not fixed.24

The mechanism for proving tractability or in-tractability is the same in statistical mechanics asit is in computational complexity: polynomial re-duction. Francisco Barahona, for example, ap-plies a reduction of the NP-hard problem MaxCut to the Ising spin glass in 3d to proof the lat-ter’s NP hardness.22 A reduction of the planarIsing model to Minimum Weight Matching, onthe other hand, proofs the tractability of the for-mer because we can solve Minimum WeightMatching in polynomial time.

In our classification of spin systems, the na-ture of the couplings is not significant. A frus-trated, glassy system with random couplings Ji,jof both signs is in the same class as the homo-geneous ferromagnet with Ji,j ≡ J > 0 as long asthe underlying graph G is the same. In the 1Dexample, we did not discriminate these cases:they are both polynomial. This situationchanges, of course, if we consider the groundstates rather than the complete partition func-tion. Here, the nature of the couplings matters alot: finding the ground states in pure ferromag-netic systems is trivial on all lattices, whereas itis NP hard for glassy systems with positive andnegative couplings on nonplanar lattices.22

G

Z G e e eJ H Hi j i j i i( ˆ ) [ ],= +( )− − ∑ − ∑ ∑∑ β σ σ β σ β σ

σ

G

G

Z G n E ek

E

E

k

k

( ) ( ) ,= −∑ β

Z

e H H e H

NN N

J H

= + =

± −

+ − ±

−

λ λ λ

β β ββ β

with

cosh( ) cosh ( ) sinh( ) ,2 22 2

Z e e

e eN

J H J

J J Hk

Nk k

k k( )

( )

( )ring Tr=

+ −

− −=

∏β β

β β1

N

σσσ σ1 2

3

JNJ

1J

2

Figure 8. One-dimensional Ising spin glass with periodic boundaryconditions. We can calculate this system’s partition sum in polynomial time.


MAY/JUNE 2002 45

We can classify many other problems arisingin statistical physics according to the computa-tional complexity to evaluate their partitionfunction.26 We can also evaluate all the prob-lems known to have an exact solution27 in poly-nomial time. Problems that are #P complete,however, are unlikely to be exactly solvable.Anyone looking for an exact solution of such aproblem should keep in mind that he or she issimultaneously attacking TSP, Hamiltonian Cy-cle, SAT, and all the other NP-hard problems.In statistical mechanics, the focus is on resultsfor the thermodynamic limit N → ∞ rather thanfor finite systems, however. It is not clear howmuch of the “hardness” survives in this limit.

Probabilistic analysis of combinatorial problemsWe can formally consider problems from

combinatorial optimization as models in statisti-cal mechanics. The cost function is renamedHamiltonian, random instances are samples ofquenched disorder, and the formal model’sground states correspond to the solutions of theoptimization problems. In this way, methods de-veloped in the framework of statistical mechan-ics of disordered systems become powerful toolsfor the probabilistic analysis of combinatorialproblems.28

Researchers have applied statistical mechan-ics methods, for example, to the TSP,29,30 GraphPartitioning,31 and k-SAT.32 A particularly niceexample of this approach is the analysis of As-signment (also called Bipartite Matching): Givenan N × N matrix with nonnegative entries ai,j ≥0, find

(20)

where the minimum is taken over all permuta-tions σ of (1, ..., N).

A probabilistic analysis aims at calculating av-erage properties for an ensemble of random in-stances, the canonical ensemble being randomnumbers ai,j drawn independently from a com-mon probability density ρ(a). Using the replicamethod from statistical physics, Marc Mézardand Giorgio Parisi33 found (among other things)

, (21)

where ⟨•⟩ denotes averaging over the ai,j. DavidAldous recently presented34 a rigorous proof ofEquation 21, which represents one of the rarecases where rigorous methods have confirmed areplica result.

Some years after the replica solution, Parisirecognized that for exponentially distributedmatrix elements (ρ(a) = e–a), the average optimafor N = 1 and N = 2 are

. (22)

From this and the fact that we can write thereplica result for N→∞ as

(23)

he conjectured35 that the average optimum forfinite systems is

. (24)

Parisi’s conjecture is supported by numericalsimulations, but no formal proof has been founddespite some efforts.36

Equations 22 and 24 only hold for ρ(a) = e–a,whereas Equation 21 is valid for all densitieswith ρ(a → 0) = 1. For the uniform density on[0,1], the first terms are

. (25)

(If you can you guess the expression for general,finite N in this case, please send me an email.)

Sometimes a statistical mechanics analysis notonly yields exact analytical results but also re-veals features that are important to design andunderstand algorithms. A recent example is theanalysis of the Davis-Putnam algorithm forSAT.37 Another example is given by the numberpartitioning problem (NPP), an NP-hard opti-mization problem.38 “Physical” reasoning has ledto the conjecture that for this problem, noheuristic algorithm can outperform simple ran-dom search.39

Phase transitions in computational complexityThe theory of computational complexity is

based entirely on worst-case analysis. An algo-rithm could require exponential time on patho-logical instances only. A famous example is thesimplex method for linear programming. De-spite its exponential worst-case complexity, it isused in many applications to solve really largeproblems. Apparently the instances that triggerexponential running time do not appear underpractical conditions.

Linear programming is in P thanks to the El-

E E1 2

12

2330

* *= =

E

kNk

N* =

=∑ 1

21

π 2

216

1==

∞

∑ kk

E E1 2 21 1

12

* *= = +

lim *

N NE→ ∞

= π 2

6

E aN i i

i

N*

, ( )min ,==

∑σ σ1



lipsoid algorithm,5 but similar scenarios are ob-served for NP-hard problems. Take 3-SAT, forexample. When generating random instanceswith N variables and M clauses and feeding themto a smart but exhaustive algorithm, we observepolynomial running time unless the ratio M/Nis carefully adjusted. If M/N is too low, the prob-lem is underconstrained—that is, it has many sat-isfying solutions, and a clever algorithm will findone of these quickly. If M/N is too large, theproblem is overconstrained—that is, it has manycontradictory constraints, which, again, a cleveralgorithm will discover quickly.40 The real hardinstances are generated for ratios α = M/N closeto a critical value αc.41

The transition from underconstrained to over-constrained formulas in 3-SAT bears the hall-marks of a phase transition in physical systems.The control parameter is α; the order parame-ter is the probability of the formula being satis-fiable. Similar phase transitions do occur in var-ious other decision or optimization problems,and mathematical methods from statistical me-chanics have successfully been used for theiranalysis.

Reading up to this point hopefully hasconvinced you that there are some in-teresting links between physics and thetheory of computational complexity.

In fact, mathematicians, computer scientists, andphysicists have just started interdisciplinary workin this field. Tools and notions from statisticalmechanics might shed more light on the typicalcase complexity of problems and might help toimprove heuristic algorithms. Quantum com-puting may even turn intractable problems intotractable ones some day. A polynomial time al-gorithm for an NP-complete problem would bea real breakthrough—a theoretical breakthroughfirst of all, but in the far future, quantum com-puters might be available as hardware. If this re-ally happens, computational complexity andphysics will no longer be considered separatefields.

References1. D. Coppersmith and S. Winograd, “Matrix Multiplication via

Arithmetic Progressions,” J. Symbolic Computation, vol. 9, no. 3,1990, pp. 251–280.

2. T.H. Cormen, C.E. Leiserson, and R.L. Rivest, Introduction to Al-gorithms: The MIT Electrical Engineering and Computer Science Se-ries, MIT Press, Cambridge, Mass., 1990.

3. B. Bollobás, Modern Graph Theory, Graduate Texts in Mathe-matics, vol. 184, Springer-Verlag, Berlin, 1998.

4. G. Reinelt, “The Traveling Salesman,” Computational Solutionsfor TSP Applications, Lecture Notes in Computer Science, vol. 840,Springer-Verlag, Berlin, 1994.

5. C.H. Papadimitriou and K. Steiglitz, Combinatorial Optimization,Prentice-Hall, Englewood Cliffs, N.J., 1982.

6. L. Euler, “Solutio Problematis Ad Geometrian Situs Pertinentis,”Commetarii Academiae Scientiarum Imperialis Petropolitanae, vol.8, 1736, pp. 128–140.

7. H.R. Lewis and C.H. Papadimitriou, “The Efficiency of Algo-rithms,” Scientific American, vol. 109, no. 1, Jan. 1978, pp.96–109.

8. V. Kumar, “Algorithms for Constraint Satisfaction Problems: ASurvey,” AI Magazine, vol. 13, no. 1, 1992, pp. 32–44; ftp://ftp.cs.umn.edu/dept/users/kumar/csp-aimagazine.ps.

9. B. Aspvall, M.F. Plass, and R.E. Tarjan, “A Linear-Time Algorithmfor Testing the Truth of Certain Quantified Boolean Formulas,”Information Processing Letters, vol. 8, no. 3, 1979, pp. 121–123.

10. D.S. Johnson and C.H. Papadimitriou, “Computational Com-plexity,” The Traveling Salesman Problem, Lawler et al., eds.,1985, pp. 37–85.

11. V.R. Pratt, “Every Prime has a Succinct Certificate,” SIAM J. Com-puting, vol. 4, no. 3, 1975, pp. 214–220.

12. G. Ausiello et al., Complexity and Approximation, Springer-Verlag,Berlin, 1999.

13. R.M. Karp, “Complexity of Computer Computations,” Reducibil-ity Among Combinatorial Problems, R.E. Miller and J.W. Thatcher,eds., Plenum Press, New York, 1972, pp. 85–103.

14. S. Cook, “The Complexity of Theorem Proving Procedures,” Proc.3rd Ann. ACM Symp. Theory of Computing, ACM Press, New York,1971, pp. 151–158.

15. R.E. Ladner, “On the Structure of Polynomial Time Reducibility,”J. ACM, vol. 22, no. 1, 1975, pp. 155–171.

16. L.G. Valiant, “The Complexity of Computing the Permanent,”Theoretical Computer Science, vol. 8, 1979, pp. 189–201.

17. R. Feynman, “Simulating Physics with Computers,” Int’l J. Theo-retical Physics, vol. 21, nos. 6 and 7, 1982, pp. 467–488.

18. P.W. Shor, “Polynomial-Time Algorithms for Prime Factorizationand Discrete Logarithms on a Quantum Computer,” SIAM J.Computing, vol. 26, no. 5, 1997, pp. 1484–1509.

19. B. Hayes, “Computing Science: The World in a Spin,” AmericanScientist, vol. 88, no. 5, Sept./Oct. 2000, pp. 2384–388; www.amsci.org/amsci/issues/comsci00/compsci2000-09.html.

20. F. Jaeger, D.L. Vertigan, and D.J.A. Welsh, “On the Computa-tional Complexity of the Jones and Tutte Polynomials,” Mathe-matical Proc. Cambridge Philosophical Soc., vol. 108, 1990, pp.35–53.

21. S. Istrail, “Statistical Mechanics, Three-Dimensionality and NP-Completeness,” Proc. 31st ACM Ann. Symp. Theory of Computing(STOC 2000), ACM Press, New York, 2000, pp. 87–96.

22. F. Barahona, “On the Computational Complexity of Ising SpinGlass Models,” J. Physics A: Mathematical and General, vol. 15,vol. 10, 1982, pp. 3241–3253.

23. L. Onsager, “Crystal Statistics I: A Two-Dimensional Model withan Order-Disorder Transition,” Physical Rev., vol. 65, 1944, pp.117–149.

24. T. Regge and R. Zecchina, “Combinatorial and Topological Ap-proach to the 3D Ising Model,” J. Phys. A, vol. 33, 2000, pp.741–761.

25. A. Galluccio, Martin Loebl, and Jan Vondrák, “New Algorithm forthe Ising Problem: Partition Function for Finite Lattice Graphs,”Physical Rev. Letters, vol. 84, no. 26, 2000, pp. 5924–5927.

26. D.J.A. Welsh, “The Computational Complexity of Some Classi-cal Problems from Statistical Physics,” Disorder in Physical Sys-


tems, G.R. Grimmett and D.J.A. Welsh, eds., Clarendon Press,Oxford, 1990, pp. 307–321.

27. R.J. Baxter, Exactly Solved Models in Statistical Mechanics, Acade-mic Press, San Diego, 1982.

28. M. Mézard, G. Parisi, and M.A. Virasoro, Spin Glass Theory andBeyond, World Scientific, Singapore, 1987.

29. M. Mézard and G. Parisi, “Mean-Field Equations for the Match-ing and the Traveling Salesman Problems,” Europhys. Letters, vol.2, no. 12, Dec. 1986, pp. 913–918.

30. W. Krauth and M. Mézard, “The Cavity Method and the Travel-ing-Salesman Problem,” Europhys. Lett., vol. 8, no. 3, 1989, pp.213–218.

31. Y. Fu and P.W. Anderson, “Application of Statistical Mechanics toNP-Complete Problems in Combinatorial Optimization,” J.Physics A: Mathematical and General, vol. 19, 1986, pp.1605–1620.

32. R. Monasson and Riccardo Zecchina, “Statistical Mechanics ofthe Random k-Satisfiability Model,” Phys. Rev. E, vol. 56, no. 2Aug. 1997, pp. 1357–1370.

33. M. Mézard and G. Parisi, “Replicas and Optimization,” J. PhysiqueLetters, vol. 46, 1985, pp. L771–L778.

34. D. J. Aldous, “The ζ(2) Limit in the Random Assignment Prob-lem,” Random Structures & Algorithms, vol. 18, no. 4, July 2001,pp. 381–418.

35. G. Parisi, A Conjecture on Random Bipartite Matching, 1998;http://xxx.lanl.gov/PS_cache/cond-mat/pdf/9801/9801176.pdf.

36. D. Coppersmith and G. Sorkin, “Constructive Bounds and ExactExpectations for the Random Assignment Problem,” RandomStructures and Algorithms, vol. 15, no. 2, 1999, pp. 113–144.

37. S. Cocco and R. Monasson, “Statistical Physics Analysis of theComputational Complexity of Solving Random Satisfiability Prob-lems using Backtrack Algorithms,” European Physics J. B, vol. 22,

2001, pp. 505–531.

38. B. Hayes, “Computing Science: The Easiest Hard Problem,”Amer-ican Scientist, vol. 90, no. 2, Mar./Apr. 2002, pp. 113–117;www.amsci.org/amsci/issues/comsci02/compsci2002-03.html.

39. S. Mertens, “Random Costs in Combinatorial Optimization,”Physical Rev. Letters, vol. 84, no. 7, Feb. 2000, pp. 1347–1350.

40. B. Hayes, “Computing Science: Can’t Get No Satisfaction,” Amer-ican Scientist, vol. 85, no. 2, Mar./Apr. 1997, pp. 108–112;www.amsci.org/amsci/issues/Comsci97/compsci9703.html.

41. R. Monasson et al., “Determining Computational Complexityfrom Characteristic ‘Phase Transitions,’” Nature, vol. 400, July1999, pp. 133–137.

Stephan Mertens is a teaching and research assistant atthe Institute for Theoretical Physics, University of Magde-burg. His fields of research are statistical mechanics ofhard optimization problems, phase transitions in com-putational complexity, and parallel algorithms. He runsthe Institute’s computing machinery, including a 156-node Beowulf cluster. He received his PhD in theoreticalphysics from Göttingen University. Contact him at Otto-von-Guericke Univ., Inst. f. Theor. Physik, Otto-von-Gu-ericke Univ., PF 4120, D-39016 Magdeburg, Germany;[email protected].

For more information on this or any other computingtopic, please visit our Digital Library at http://computer.org/publications/dlib.

What is Computing inScience & Engineeringmagazine?

Computing in Science & Engineering is a joint publication of the IEEE Computer Society and the American Institute of Physics

www.aip.orghttp://computer.org

®.

May/June 2001

FOR

Electrical Engineers

Physicists

Chemists

and others...

SUBSCRIBE TODAY!Visit www.computer.org/cise or http://ojps.aip.org/cise

CiSE is a peer-reviewed, joint

publication of the IEEE Computer Society and

the American Institute of Physics. It represents

the merger of two first-rate scientific publica-

tions—AIP’s Computers in Physics and IEEE

Computational Science & Engineering.

Now electrical engineers, physicists, chemists,

and others have a magazine that covers a

broad range of topics, emphasizing the com-

mon techniques and practical lessons that are

portable from one area of CSE to another. CiSE

is the leading interdisciplinary forum for those

who call themselves computational scientists

or engineers, or who have an interest in the

subject. Specialists from many areas find it

highly readable and accessible.


Computational complexity for physicists - Computing in...

Documents

Transcript of Computational complexity for physicists - Computing in...