LP Duality

John E. Mitchell

Department of Mathematical Sciences

RPI, Troy, NY 12180 USA

January 2016

Mitchell LP Duality 1 / 26

Outline

1 Weak duality

2 Strong duality

3 Complementary slackness

4 Farkas Lemma

5 Visualizing the Farkas Lemma

Mitchell LP Duality 2 / 26

Outline

LP dual pair

We work with the standard form linear program and its dual:

minx2IRn cT x maxy2IRm bT ysubject to Ax = b (P) subject to AT y c (D)

x � 0

where c 2 IRn, b 2 IRm, and A 2 IRm⇥n. The results extend to any

primal-dual pair, including

minx2IRn cT x maxy2IRm bT ysubject to Ax � b (P) subject to AT y c (D)

x � 0 y � 0

Mitchell LP Duality 3 / 26

Weak duality

Outline

1 Weak duality

2 Strong duality

3 Complementary slackness

4 Farkas Lemma

5 Visualizing the Farkas Lemma

Mitchell LP Duality 4 / 26

Weak duality

Weak duality

Theorem

Weak duality: If x is feasible in (P) and y is feasible in (D) thencT x � bT y.

Proof.

We have

bT y = (Ax)T y = xT AT y xT c = cT x

where the inequality follows because x � 0 and AT y c.

Mitchell LP Duality 5 / 26

Weak duality

Weak duality

Theorem

Weak duality: If x is feasible in (P) and y is feasible in (D) thencT x � bT y.

Proof.

We have

bT y = (Ax)T y = xT AT y xT c = cT x

where the inequality follows because x � 0 and AT y c.

Mitchell LP Duality 5 / 26

Arab, 5 7 0 AtfE c

Weak duality

Sufficient optimality criterion

As an immediate consequence of the weak duality theorem, we have

the following sufficient condition for optimality:

Theorem

Sufficient optimality criterion: If x is primal feasible and y is dualfeasible and cT x = bT y then x is optimal in (P) and y is optimal in (D).

Mitchell LP Duality 6 / 26

Strong duality

Outline

1 Weak duality

2 Strong duality

3 Complementary slackness

4 Farkas Lemma

5 Visualizing the Farkas Lemma

Mitchell LP Duality 7 / 26

Strong duality

Strong duality

This condition is also necessary:

Theorem

Strong duality theorem: If (P) is feasible with a finite optimal valuethen (D) is also feasible. Further, there exist optimal solutions x⇤ andy⇤ with cT x⇤ = bT y⇤.

We will prove this theorem later. In particular, we will use the simplex

algorithm to show how a dual optimal solution can be constructed with

cT x⇤ = bT y⇤.

Mitchell LP Duality 8 / 26

Strong duality

Possible states of an optimization problem

A general optimization problem can be in one of four possible states:

Infeasible

Feasible with an unbounded optimal value.

Feasible with a finite optimal value that is attained.

Feasible with a finite optimal value that is not attained.

It follows from the strong duality theorem that this last case is not

possible for a linear program. The strong duality theorem also implies

that the state of a linear program tells us the state of its dual linear

program. So a primal-dual pair can only be in one of 4 possible states,

rather than 4 ⇥ 4 = 16 possible states.

Mitchell LP Duality 9 / 26

Strong duality

Possible states of an optimization problem

A general optimization problem can be in one of four possible states:

Infeasible

Feasible with an unbounded optimal value.

Feasible with a finite optimal value that is attained.

Feasible with a finite optimal value that is not attained.

It follows from the strong duality theorem that this last case is not

possible for a linear program. The strong duality theorem also implies

that the state of a linear program tells us the state of its dual linear

program. So a primal-dual pair can only be in one of 4 possible states,

rather than 4 ⇥ 4 = 16 possible states.

Mitchell LP Duality 9 / 26

Strong duality

Possible states of an optimization problem

A general optimization problem can be in one of four possible states:

Infeasible

Feasible with an unbounded optimal value.

Feasible with a finite optimal value that is attained.

Feasible with a finite optimal value that is not attained.

It follows from the strong duality theorem that this last case is not

possible for a linear program. The strong duality theorem also implies

that the state of a linear program tells us the state of its dual linear

program. So a primal-dual pair can only be in one of 4 possible states,

rather than 4 ⇥ 4 = 16 possible states.

Mitchell LP Duality 9 / 26

Strong duality

Possible states of an optimization problem

A general optimization problem can be in one of four possible states:

Infeasible

Feasible with an unbounded optimal value.

Feasible with a finite optimal value that is attained.

Feasible with a finite optimal value that is not attained.

It follows from the strong duality theorem that this last case is not

possible for a linear program. The strong duality theorem also implies

that the state of a linear program tells us the state of its dual linear

program. So a primal-dual pair can only be in one of 4 possible states,

rather than 4 ⇥ 4 = 16 possible states.

Mitchell LP Duality 9 / 26

r u i n e - ×i t . x 2 0

I

Strong duality

Possible states of an LP pair

In particular, we have the following consequence of the strong duality

theorem:

Theorem

The following are mutually exclusive and exhaustive possibilities for(P) and (D):

(P) and (D) are both infeasible.(D) is infeasible and (P) has an unbounded optimal value.(P) is infeasible and (D) has an unbounded optimal value.Both (P) and (D) are feasible, and they have the same optimalvalue.

Mitchell LP Duality 10 / 26

Strong duality

Possible states of an LP pair

In particular, we have the following consequence of the strong duality

theorem:

Theorem

The following are mutually exclusive and exhaustive possibilities for(P) and (D):

(P) and (D) are both infeasible.(D) is infeasible and (P) has an unbounded optimal value.(P) is infeasible and (D) has an unbounded optimal value.Both (P) and (D) are feasible, and they have the same optimalvalue.

Mitchell LP Duality 10 / 26

Strong duality

Possible states of an LP pair

In particular, we have the following consequence of the strong duality

theorem:

Theorem

The following are mutually exclusive and exhaustive possibilities for(P) and (D):

(P) and (D) are both infeasible.(D) is infeasible and (P) has an unbounded optimal value.(P) is infeasible and (D) has an unbounded optimal value.Both (P) and (D) are feasible, and they have the same optimalvalue.

Mitchell LP Duality 10 / 26

Strong duality

Possible states of an LP pair

In particular, we have the following consequence of the strong duality

theorem:

Theorem

The following are mutually exclusive and exhaustive possibilities for(P) and (D):

(P) and (D) are both infeasible.(D) is infeasible and (P) has an unbounded optimal value.(P) is infeasible and (D) has an unbounded optimal value.Both (P) and (D) are feasible, and they have the same optimalvalue.

Mitchell LP Duality 10 / 26

Strong duality

Possible states of an LP pair

In particular, we have the following consequence of the strong duality

theorem:

Theorem

The following are mutually exclusive and exhaustive possibilities for(P) and (D):

(P) and (D) are both infeasible.(D) is infeasible and (P) has an unbounded optimal value.(P) is infeasible and (D) has an unbounded optimal value.Both (P) and (D) are feasible, and they have the same optimalvalue.

Mitchell LP Duality 10 / 26

Complementary slackness

Outline

1 Weak duality

2 Strong duality

3 Complementary slackness

4 Farkas Lemma

5 Visualizing the Farkas Lemma

Mitchell LP Duality 11 / 26

Complementary slackness

Complementary slackness

Theorem

A pair of primal and dual feasible solutions are optimal to theirrespective problems in a primal-dual pair of LPs if and only if

whenever these variables make a slackvariable in one problem strictly positivethe value of the associated nonnegativevariable in the other is zero.

9>>=

>>;

complementaryslackness

Mitchell LP Duality 12 / 26

m m × ,-12×12+34, (p)Optimal:

S t . × , -1×2++3=1 ×*=(1,0,O)x i2 0 Value : I

5 1=0m a x ys e . y@tight.

Slacker

y e a(D) Optimal:

yes}slack>o y-4=1,f - O , value: 1 .

y free s , >0

Complementary,lactic,,: s , >o , s , >oS i x t o

⇒ x x x ,-Oft i

Complementary slackness

Proof of complementary slackness

We prove this for the standard pair (P) and (D). We can define the

vector of dual slacks s = c � AT y 2 IRn for any y 2 IRm. Note that the

duality gap is

cT x � bT y = cT x � (Ax)T y = cT x � xT AT y = cT x � (AT y)T x

= (c � AT y)T x = sT x =nX

i=1

sixi

for any y 2 IRm and x 2 IRn with Ax = b.

Mitchell LP Duality 13 / 26

Complementary slackness

Proof of complementary slackness continued

Note that if x and y are feasible in their respective problems then x � 0

and s � 0, so sT x � 0.

If the points are optimal then cT x � bT y = 0 soPn

i=1sixi = 0, so each

component sixi = 0, since they must all be nonnegative. So either

si = 0 or xi = 0 for each component i . This is complementary

slackness.

If complementary slackness holds then either xi = 0 or si = 0 for each

component i , so sT x = 0 so cT x = bT y so the points are optimal. ⇤

Mitchell LP Duality 14 / 26

Complementary slackness

Complementary slackness for standard primal-dual

pair

Exercise:

Consider the primal-dual pair

minx cT x maxy bT ysubject to Ax � b (P) subject to AT y c (D)

x � 0 y � 0

and let s = c � AT y , w = Ax � b. Show cT x � bT y = xT s + yT w .

Mitchell LP Duality 15 / 26

T TExisi S yi w i

Farkas Lemma

Outline

1 Weak duality

2 Strong duality

3 Complementary slackness

4 Farkas Lemma

5 Visualizing the Farkas Lemma

Mitchell LP Duality 16 / 26

Farkas Lemma

Farkas Lemma

Theorem

Farkas Lemma: Let A 2 IRm⇥n and b 2 IRm. Exactly one of thefollowing two systems has a solution:

(I): Ax = b, x � 0. (II): AT y 0, bT y > 0.

Mitchell LP Duality 17 / 26

Farkas Lemma

Farkas Lemma

Theorem

Farkas Lemma: Let A 2 IRm⇥n and b 2 IRm. Exactly one of thefollowing two systems has a solution:

(I): Ax = b, x � 0. (II): AT y 0, bT y > 0.

Mitchell LP Duality 17 / 26

Farkas Lemma

Proof of Farkas Lemma

Proof.

Prove using LP duality. Set up the primal-dual pair (P) and (D) with

c = 0. Note that y = 0 is feasible in (D), so (D) either has finite

optimal value or unbounded optimal value. Note also that (P) is either

infeasible or has optimal value equal to zero.

If (I) is consistent then (P) has optimal value 0 so (D) also has optimal

value 0, so there is no y 2 IRm with both AT y 0 and bT y > 0, so (II)is inconsistent.

If (I) is inconsistent then (P) is infeasible so (D) has unbounded

optimal value, so there exists y with AT y 0 and bT y > 0, so (II) is

consistent.

Mitchell LP Duality 18 / 26

( I ) Axeb.→ O (II) A' je 0 ,big>o(p) m i n O x (D) m a x bey

s e . Ax-b s - t . AlysO÷( I ) consistent ⇒ (P)feasible,optimal valueof O

⇒ (D)feasible, optimalvalue0 ⇒ ( I l ) inconsistent.

-( I ) inconsistent ⇒ (P)

infeasible

⇒ either: (D) infeasible-(D)alwan,feasible:

° ' : ( D ) unboundedt a key-OER"

⇒ (D) unbounded ⇒ I T will Atyeoandbig>

o .⇒ ( I )consistent.

Visualizing the Farkas Lemma

Outline

1 Weak duality

2 Strong duality

3 Complementary slackness

4 Farkas Lemma

5 Visualizing the Farkas Lemma

Mitchell LP Duality 19 / 26

Visualizing the Farkas Lemma

The columns of A

Let

A =

1 3 2

4 2 1

�.

For what vectors b 2 IR2 is System I consistent?

For what b 2 IR2 is System II consistent?

Mitchell LP Duality 20 / 26

Visualizing the Farkas Lemma

Consistency of System I

For System I, we require b = Ax for some x � 0.

We can express Ax as follows:

Ax =

1 3 2

4 2 1

�2

4x1

x2

x3

3

5 =

x1 + 3x2 + 2x3

4x1 + 2x2 + x3

�

= x1

1

4

�+ x2

3

2

�+ x3

2

1

�

Thus, System I is consistent if and only if b is a nonnegative linearcombination of the columns of A. This is a cone in IRm.

Mitchell LP Duality 21 / 26

Visualizing the Farkas Lemma

Nonnegative combinations of the columns of A

b1

b2

0

4

8

4 8

(1,4)

(3,2)

(2,1)

A =

1 3 2

4 2 1

�I: Ax = b, x � 0

Mitchell LP Duality 22 / 26

Visualizing the Farkas Lemma

Nonnegative combinations of the columns of A

b1

b2

0

4

8

4 8

(1,4)

(3,2)

(2,1)

A =

1 3 2

4 2 1

�I: Ax = b, x � 0

x =

2

40.5

0

0

3

5

Ax =

0.5

2

�(0.5,2)

Mitchell LP Duality 22 / 26

Visualizing the Farkas Lemma

Nonnegative combinations of the columns of A

b1

b2

0

4

8

4 8

(1,4)

(3,2)

(2,1)

A =

1 3 2

4 2 1

�I: Ax = b, x � 0

x =

2

4t0

0

3

5

Ax =

t

4t

�

t � 0

Mitchell LP Duality 22 / 26

Visualizing the Farkas Lemma

Nonnegative combinations of the columns of A

b1

b2

0

4

8

4 8

(1,4)

(3,2)

(2,1)

A =

1 3 2

4 2 1

�I: Ax = b, x � 0

x =

2

40

0

2

3

5

Ax =

4

2

�

(4,2)

Mitchell LP Duality 22 / 26

Visualizing the Farkas Lemma

Nonnegative combinations of the columns of A

b1

b2

0

4

8

4 8

(1,4)

(3,2)

(2,1)

A =

1 3 2

4 2 1

�I: Ax = b, x � 0

x =

2

40

0

s

3

5

Ax =

2s

s

�

s � 0

Mitchell LP Duality 22 / 26

Visualizing the Farkas Lemma

Nonnegative combinations of the columns of A

b1

b2

0

4

8

4 8

(1,4)

(3,2)

(2,1)

A =

1 3 2

4 2 1

�I: Ax = b, x � 0

x =

2

40.51.5

2

3

5

Ax =

9

7

�

(9,7)

Mitchell LP Duality 22 / 26

Visualizing the Farkas Lemma

Nonnegative combinations of the columns of A

b1

b2

0

4

8

4 8

System I

consistent

for b here

(1,4)

(3,2)

(2,1)

A =

1 3 2

4 2 1

�I: Ax = b, x � 0

Mitchell LP Duality 22 / 26

Visualizing the Farkas Lemma

A polar cone

System II considers the y 2 IRm satisfying AT y 0. We have:

{y 2 IRm : AT y 0} =

8<

:y 2 IR2 :

2

41 4

3 2

2 1

3

5

y1

y2

�

2

40

0

0

3

5

9=

;

=

(y 2 IR2 :

1

4

�T y1

y2

� 0,

3

2

�T y1

y2

� 0,

2

1

�T y1

y2

� 0

)

so we consider the set of vectors y that make an angle of at least ⇡/2

with each column of A. This is another cone in IRm.

If b is a nonzero vector in this cone then one solution to System II is to

take y = b, since then bT y = bT b > 0.

Mitchell LP Duality 23 / 26

f.i t :#I

Visualizing the Farkas Lemma

Picturing the polar cone

y1

y2

0

4

�4

�6

(1,4)

(3,2)

(2,1)

II: AT y 0, bT y > 0

A =

1 3 2

4 2 1

�

Mitchell LP Duality 24 / 26

Visualizing the Farkas Lemma

Picturing the polar cone

y1

y2

0

4

�4

�6

y1 + 4y2 0

(1,4)

(3,2)

(2,1)

II: AT y 0, bT y > 0

A =

1 3 2

4 2 1

�

Mitchell LP Duality 24 / 26

Visualizing the Farkas Lemma

Picturing the polar cone

y1

y2

0

4

�4

�6

2y1 + y2 0

(1,4)

(3,2)

(2,1)

II: AT y 0, bT y > 0

A =

1 3 2

4 2 1

�

Mitchell LP Duality 24 / 26

Visualizing the Farkas Lemma

Picturing the polar cone

y1

y2

0

4

�4

�6

3y1 + 2y2 0

(1,4)

(3,2)

(2,1)

II: AT y 0, bT y > 0

A =

1 3 2

4 2 1

�

Mitchell LP Duality 24 / 26

Visualizing the Farkas Lemma

Picturing the polar cone

y1

y2

0

4

�4

�6

y1 + 4y

2 = 0

2y1 +

y2=

0

{y : AT y 0}

y1 + 4y2 0

3y1 + 2y2 0

2y1 + y2 0

(1,4)

(3,2)

(2,1)

II: AT y 0, bT y > 0

A =

1 3 2

4 2 1

�

Mitchell LP Duality 24 / 26

Visualizing the Farkas Lemma

Picturing the polar cone

y1

y2

0

4

�4

�6

y1 + 4y

2 = 0

2y1 +

y2=

0

{y : AT y 0}

y1 + 4y2 0

3y1 + 2y2 0

2y1 + y2 0

System II

consistent:

take y = b

(1,4)

(3,2)

(2,1)

II: AT y 0, bT y > 0

A =

1 3 2

4 2 1

�

Mitchell LP Duality 24 / 26

IE

Visualizing the Farkas Lemma

Other choices of b

If b is in neither cone then there is a point y on the boundary of the

second cone that makes positive inner product with b. This y satisfies

the conditions of System II.

We can graph the problem in IRm. The columns of A are points in IRm,

and the two cones are subsets of IRm.

Mitchell LP Duality 25 / 26

Visualizing the Farkas Lemma

The cones

b1

b2

0

4

�4

4�6

System I

consistent

y1 + 4y

2 = 0

2y1 +

y2=

0

{y : AT y 0}

System II

consistent:

take y = b

(1,4)

(3,2)

(2,1)

Mitchell LP Duality 26 / 26

Visualizing the Farkas Lemma

The cones

b1

b2

0

4

�4

4�6

System I

consistent

y1 + 4y

2 = 0

2y1 +

y2=

0

{y : AT y 0}

System II

consistent:

take y = b

System II

consistent:

take y = (�4, 1)so bT y > 0

(-4,1)

(1,4)

(3,2)

(2,1)

Mitchell LP Duality 26 / 26

• b

¥-97.b

Visualizing the Farkas Lemma

The cones

b1

b2

0

4

�4

4�6

System I

consistent

y1 + 4y

2 = 0

2y1 +

y2=

0

{y : AT y 0}

System II

consistent:

take y = b

System II

consistent:

take y = (�4, 1)so bT y > 0

(-4,1)

System II

consistent:

take y = (1,�2)so bT y > 0

(1,-2)

(1,4)

(3,2)

(2,1)

Mitchell LP Duality 26 / 26