Problem Complexity and NP-Complete Problems

Presented by Ming-Tsung Hsu

“I can’t find an efficient algorithm, I guess I’m just too dumb”

“I can’t find an efficient algorithm, because no such algorithm is possible”

“I can’t find an efficient algorithm, but neither can all these famous people”

Problem Complexity

Overview

Complexity theory• Part of the theory of computation• Dealing with the resources required to solve a

given problem– time (how many steps it takes to solve a

problem) – space (how much memory it takes)

• Computational difficulty of computable functions

Relations between Problems, Algorithms, and Programs

Problem

Algorithm Algorithm

Program ProgramProgram Program

. . . .

. . . . . . . .

•A single "problem" is an entire set of related questions, where each question is a finite-length string •A particular question is called an instance

Time Complexity

• Number of steps that it takes to solve an instance of the problem – Function of the size of the input – Using the most efficient algorithm

• Exact number of steps will depend on exactly what machine or language is being used– To avoid that problem, generally use Big O

notation

Time Complexity (cont’d)

• Worst-case– an upper bound on the running time for any

input• Average-case

– we shall assume that all inputs of a given size are equally likely

• Best-case– to get the lower bound

Time Complexity (cont’d)

• Sequential search in a list of size n– worst-case : n times– best-case : 1 times– average-case :

n

i

ni

n 1 2

)1(1

Asymptotic Notation - O-notation (Big O, Order of)

• Asymptotic upper bound• Definition: For a given

function g(n), denoted by O(g(n)) is the set of functions: O(g(n)) = {there exist positive constants c and n0 such that f(n) <= cg(n) for all n >= n0}

Asymptotic Notation - Ω-notation

• Asymptotic lower bound• Definition: For a given

function g(n), denoted by Ω(g(n)) is the set of functions:(g(n)) = {there exist positive constants c and n0

such that f(n) >= cg(n) for all n >= n0}

Asymptotic Notation – -notation

• Definition: For a given function g(n), denoted by (g(n)) is the set of functions: (g(n)) = {there exist positive constants c1, c2, and n0 such that c1g(n) <= f(n) <= c2g(n) for all n >= n0}

Cost and Complexity

• Algorithm complexity can be expressed in Order notation, e.g. “at what rate does work grow with N?”:– O(1) Constant– O(logN) Sub-linear– O(N) Linear– O(NlogN) Nearly linear– O(N2) Quadratic– O(XN) Exponential

• But, for a given problem, how do we know if a better algorithm is possible?

112/04/20

Practical Complexities

109 instructions/second computer

n n nlogn n2 n3

1000 1mic 10mic 1milli 1sec

10000 10mic 130mic 100milli 17min

106 1milli 20milli 17min 32years

112/04/20

Impractical Complexities

109 instructions/second computer

n n4 n10 2n

1000 17min 3.2 x 1013

years3.2 x 10283

years

10000 116days

??? ???

10^6 3 x 10^7years

?????? ??????

112/04/20

Faster Computer Vs Better Algorithm

Algorithmic improvement more useful

than hardware improvement.

E.g. 2n to n3

The Problem of Sorting

For example, in discussing the problem of sorting:

• Algorithms:– Bubble-sort – O(N2)– Merge-sort – O(N Log N)– Quick-sort – O(N Log N)

• Can we do better than O(N Log N)? What is the problem complexity?

Algorithm vs. Problem Complexity

• Algorithm complexity is defined by analysis of an algorithm

• Problem complexity is defined by– An upper bound – defined by an

algorithm (worst case)– A lower bound – defined by a proof (A

lot of problems are still unknown)

The Upper Bound

• Defined by an algorithm• Defines that we know we can do

at least this good• Perhaps we can do better• Lowered by a better algorithm

– “For problem X, the best algorithm was O(N3), but my new algorithm is O(N2).”

The Lower Bound

• Defined by a proof• Defines that we know we can do no better

than this• It may be worse• Raised by a better proof

– “For problem X, the strongest proof showed that it required O(N), but my new, stronger proof shows that it requires at least O(N2).”

– Might get harder and harder

Upper and Lower Bounds

• The Upper bound is the best algorithmic solution that has been found for a problem– “What’s the best that we know we can do?”

• The Lower bound is the best solution that is theoretically possible.– “What cost can we prove is necessary?”

Changing the Bounds

Upper bound Lowered by betteralgorithm

Lower bound Raised by betterproof

Closed Problems

The upper and lower bounds are identical

Upper bound

Lower bound

The inherent complexity of problem

Closed Problems (cont’d)

• Better algorithms are still possible

• Better algorithms will not provide an improvement detectable by “Big O”

• Better algorithms can improve the constant costs hidden in “Big O” characterizations

Open Problems

The upper and lower bounds differ.

Upper bound Lowered by betteralgorithm

Lower bound Raised by betterproof

Unknown Who Falls Short?

Open Problems (cont’d)

• D. Harel. Algorithmics: The Spirit of Computing. Addison-Wesley, 2nd edition,1992

• “. . . if a problem gives rise to a. . . gap, the deficiency is not in the problem but in our knowledge about it. We have failed either in finding the best algorithm for it or in proving that a better one does not exist, or in both”

Examples - Closed

• Problem: Searching an unordered list of n items– Upper bound: O(n) comparisons (from linear search)– Lower bound: O(n) comparisons– No gap; so we know the problem complexity: O(n)

• Problem: Searching an ordered list of n items– Upper bound: O(log n) comparisons (from binary

search)– Lower bound: O(log n) comparisons– No gap; so we know the problem complexity: O(log n)

Examples - Closed (cont’d)

• Problem: Sorting n arbitrary elements– Upper bound: O(n × log n) comparisons (from, e.g.,

merge sort)– Lower bound: O(n × log n) comparisons– No gap; so we know the problem complexity: O(n × log

n)• Problem: Towers of Hanoi for n disks (n > 1)

– Upper bound: O(2n) moves– Lower bound: O(2n) moves– No gap; so we know the problem complexity: O(2n)

Examples - Open

• Problem: Multiplication of two integers, where n is the number of digits – Upper bound: O(nlognloglogn)– Lower bound: O(n)– There’s a gap (but only a small one)

• Problem: Finding the minimum spanning tree in a graph of n edges and m vertices.– Upper bound: O(nlogn) or O(n+mlogm)– Lower bound: O(n)– There’s a gap (but only a small one)

• Do not be fooled by the last two examples into thinking that all gaps are small!

Tractable vs. Intractable

Problems are tractable if the upper bounds and lower bounds have only polynomial factors– O (log N)– O (N)

– O (NK) where K is a constant

Problems are intractable if the upper bounds and lower bounds have an exponential factor(are solvable in theory, but can't be solved in practice)– O (N!)

– O (NN)

– O (2N)

Terminology

• Polynomial algorithms are reasonable• Polynomial problems are tractable

• Exponential algorithms are unreasonable• Exponential problems are intractable

Terminology

Tractable

Intractable

Reasonable

Unreasonable

Problems Algorithms

Polynomial

Exponential

NP-Complete Problems

Definitions of P, NP

• A decision problem is a problem where the answer is always “YES/NO”

• An optimization problem is a problem can be many possible solutions. Each solution has a value, and we wish to fond a solution with the optimal (maximum or minimum) value

• We can re-cast an optimization problem to a decision problem by using loop

• EX: Find Max Clique in a graph Is there a clique which size is k=n, n-1, …?

Definitions (cont’d)

• A deterministic algorithm is an algorithm which, in informal terms, behaves predictably. If it runs on a particular input, it will always produce the same correct output, and the underlying machine will always pass through the same sequence of states

• A non-deterministic algorithm has two stages:• 1. Guessing (in nondeterministic polynomial time

with correct guessing) Stage• 2. Verification (in deterministic polynomial time) S

tage

Definitions (cont’d)

• EX: A: (3, 11, 2, 5, 8, 16, …, 200 ), Is the “x=5” in A?• deterministic algorithm

• for i=1 to n if A(i) = x then print (i) and return truenextreturn false

• non-deterministic algorithm• j choice (1:n)

if A(j) = x then print (j); success endifprint (‘0’); failure

Certificates

• Returning true: in order to show that the solution can be made, we only have to show one solution that works– This is called a certificate.

• Returning false: in order to show that the solution cannot be made, we must test all solution.

Oracles

• If we could make the ‘right decision’ at all decision points, then we can determine whether a solution is possible very quickly!– If the found solution is valid, then True– If the found solution is invalid, then False

• If we could find the certificates quickly, NPC problems would become tractable – O(N)

• This (magic) process that can always make the right guess is called an Oracle.

Determinism vs. Nondeterminism

• Nondeterministic algorithms produce an answer by a series of “correct guesses”

• Deterministic algorithms (like those that a computer executes) make decisions based on information.

Definitions (cont’d)

• The complexity class P is the set of decision problems that can be solved by a deterministic machine (algorithm) in polynomial time

• The complexity class NP is the set of decision problems that can be solved by a non-deterministic machine (algorithm) in polynomial time

• Since deterministic algorithms are just a special case of deterministic ones P NP⊆

• The big question: Does P = NP?

Definitions (cont’d)

Problems that Cross the Line

• What if a problem has:– An exponential upper bound– A polynomial lower bound

• We have only found exponential algorithms, so it appears to be intractable.

• But... we can’t prove that an exponential solution is needed, we can’t prove that a polynomial algorithm cannot be developed, so we can’t say the problem is intractable...

Reduction

• A problem P can be reduced to another problem Q if any instance of P can be rephrased to an instance of Q, the solution to which provides a solution to the instance of P– This rephrasing is called a transformation

• If P is polynomial-time reducible to Q, we denote this P ∝ Q

• Intuitively: If P reduces in polynomial time to Q, P is “no harder to solve” than Q

• EX: Maximum ∝ Sorting but Sorting cannot be reduced to Maximum

The Boolean Satisfiability Problem (SAT)

• An instance of the problem is a Boolean expression written using only “AND”, “OR”, “NOT”, “variables”, and “parentheses”

• Question: given the expression, is there some assignment of “TRUE” and “FALSE” values to the variables will make the entire expression true?

• Clause: “OR” together a group of literals• Conjunctive normal form (CNF): formulas that are a

conjunction (AND) of clauses• SAT NP∈ O(2nm) (n: # of literals, m: # of clauses)

SAT (cont’d)

• EX: CNF: a1 V a2 V a3 (clause1)& ā1 (clause2) (0, 0, 1) satisfied& ā2 (clause3)

Cook’s Theorem

• NP = P iff SAT ∈ P• pf:

() SAT NP ∵ ∈ SAT P∈() Difficult (every NP problem can be reduced to SAT problem)

NP-Complete and NP-Hard

• A problem P is NP-Complete if P ∈ NP and every NP problem can be reduced to P– P ∈ NP and X P for all X ∝ ∈NP– If any NP-Complete problem can be solved in

polynomial time, then NP = P– SAT problem is an NP-complete problem

• A problem P is NP-Hard if every NP problem can be reduced to P– We say P is NP-Complete if P is NP-Hard

and P NP

NP-Complete

“NP-Complete” comes from: – Nondeterministic Algorithm in

Polynomial time– Complete - “Solve one, Solve them all”

There are more NP-Complete problems than provably intractable problems.

NP-Hard

“NP-Hard” comes from: – Nondeterministic Algorithm in

Polynomial time– Hard - “as "hard" as any problem in

NP”

NP-Complete and NP-Hard (cont’d)

NP NPC NPH

Example NP-Complete Problems

• Path-Finding (Traveling salesman)• Map coloring• Scheduling and Matching (bin packing)• 2-D arrangement problems• Planning problems (pert planning)• Clique

Traveling Salesman

5-Clique

Map Coloring

The Knapsack Problem

• The 0-1 knapsack problem:– A thief must choose among n items, where the ith item

worth vi dollars and weighs wi pounds

– Carrying at most W pounds, maximize value– EX: B = {13,17,19,21,24,27,36,42}, S = 93, Find

X=(X1,X2,…,X8), where Xi=0 or 1, i=1,2,3,…,8 such that ΣXiBi = S

• A variation, the fractional knapsack problem:– Thief can take fractions of items– Think of items in 0-1 problem as gold ingots, in

fractional problem as buckets of gold dust

Class Scheduling Problem

• With N teachers with certain hour restrictions M classes to be scheduled, can we:– Schedule all the classes– Make sure that no two teachers teach

the same class at the same time– No teacher is scheduled to teach two

classes at once

Proving NP-Completeness

• Show that the problem A is in NP. (i.e. Show that a certificate can be verified in polynomial time)

• Assume A is not NP complete• Show how to convert an existing NPC problem into the

problem A (in polynomial time)– P is NP-Complete– P A∝– Proving that reduction is polynomial

A is NP-Complete

• If we can do it we’ve done the proof!• If we can turn an existing NP-complete problem into our

problem in polynomial time...

3-SAT is NPC ?

• 3-SAT is NP• SAT ∝ 3-SAT (by construct sets of unsatisfiable

clauses)– Clause contains only one literal Li

• Q1 V Q2 Li V Q1 V Q2 & Ǭ1 V Q2 & Li V Ǭ1 V Q2 & Q1 V Ǭ2 & Li V Q1 V Ǭ2 & Ǭ1 V Ǭ2 & Li V Ǭ1 V Ǭ2

– Clause contains two literals Li and Lj• Q1 Li V Lj V Q1

& Ǭ1 & Li V Lj V Ǭ1

3-SAT is NPC? (cont’d)

– Clause contains three literals do nothing– Clause contains more than three literals (L1, L2, …, Ln )

– Reduction is O(mn) (m: # of clauses , n: # of literals)

Q

Q V Q

Q V Q

Q V Q

Q

3-n

n4-n

2

1

1

&

&

&

&

3

3

2

Q V L V L

Q V Q V L

Q V QV L

Q V Q V L

Q V L V L

3-nnn

n4-nn

2

1

121

1

32

34

23

&

&

&

&

3-SAT is NPC? (cont’d)

• Example:

X V X V X V X V X

X V X3 V X

X V X

X

5432

42

32

1

1&

&

&

Y V X V X &

Y V Y V X&

Y V X V X

X V X3 V X

Y V X V X

Y V X V X

Y V Y V X &

Y V Y V X &

Y V Y V X &

Y V Y V X

554

543

421

42

332

332

211

211

211

211

&

&

&

&

3-SAT is NPC? (cont’d)

L1 L2 L3 L4 L5 Y1 Y2 Y3 Y4 Y5

1 1 1 1 0 0 0 0 0 0

1 1 0 0 0 0 0 0 0 1

1 1 0 1 0 0 0 0 0 0

1 1 0 0 1 0 0 0 0 0

1 1 0 1 1 0 0 0 0 0

1 1 1 1 1 0 0 0 0 0

1 0 0 0 0 0 0 0 0 1

1 0 0 1 0 0 0 0 0 0

1 0 0 0 1 0 0 0 0 0

1 0 0 1 1 0 0 0 0 0

L1 L2 L3 L4 L5

1 1 1 1 0

1 1 0 0 0

1 1 0 1 0

1 1 0 0 1

1 1 0 1 1

1 1 1 1 1

1 0 0 0 0

1 0 0 1 0

1 0 0 0 1

1 0 0 1 1

Clique Decision Problem (CDP) is NPC

• Construct a graph G = (V,E) from such that G will have a clique of size at least m iff F is satisfiable (SAT ∝ CDP)

• Construction Rule– A literal in a clause is a vertex (node) in G– A edge is a intersection of two clauses

• An assignment of F– (a) One literal is true then the clause is true – (b) Ā & A does not exist

Add edges between any two nodes exclusive of • nodes are in the same clause and nodes are exclusive

imi

CF

1&

CDP (cont’d)

• Example: F = (a1 V a2 V a3) & (ā1 V a2 V ā3) & (ā2) and G constructed by F– F is satisfied by

(1,0,0) or (0,0,1) G has a clique of size at least 3

– m = 3 F is satisfiedā2, 3

a1, 1 ā1, 2

a2, 1

ā3, 2a3, 1

a2, 2

CDP (cont’d)

Proof: support F has m clauses( ) F is satisfiable exists an assignment

• the literals in this assignment forms a clique with size equal m (a complete subgraph of G with degree m-1)

()If we can find a clique C with size equal m• Every clause has exactly one literal in C• Any two vertices in C are jointed and each edge is

the intersection of two clauses Every clause is true by literals in C F is satisfied

Become Famous!

To get famous in a hurry, for any NP-Complete problem:– Raise the lower bound

(via a stronger proof)– Lower the upper bound

(via a better algorithm)

They’ll be naming buildings after you before you are dead!

Decidable vs. Undecidable Problems

Decidable Problems

• We now have three categories:– Tractable problems– NP-Complete problems– Intractable problems

• All of the above have algorithmic solutions, even if impractical.

Undecidable Problems

• No algorithmic solution exists– Regardless of cost– These problems aren’t computable– No answer can be obtained in finite

amount of time

The Unbounded Tiling Problem

• What if we took the tiling puzzle and removed the bounds:– For a given number (T) of different kinds

of tiles– Can we arrive at an arrangement that

will fill any size area?• By removing the bounds, this problem

becomes undecidable.

The Halting Problem

Given an algorithm A and an input I, will the algorithm reach a stopping place?

loop

exitif (x = 1)

if (even(x)) then

x <- x div 2

else

x <- 3 * x + 1

endloop• In general, we cannot solve this problem

in finite time.

Partially and Highly Undecidable

• Most undecidable problems have finite certificates. These are partially decidable.

• Some undecidable problems do not have finite certificates even with an oracle. These are called highly undecidable.

A Hierarchy of Problems

• For a given problem, is there or will there ever be an algorithmic solution?

Problem In Theory In Application Highly Undecidable NO NO Partially Undecidable NO NO Intractable YES NO Tractable YES YES

Thanks