Basics of maximization - Essential...

Mathematical Foundations -1- Basics of Maximization

© John Riley October 8, 2013

Basics of maximization A. Review: Differentiating a function of a function 2

B. Unconstrained maximization 13

C. Maximization when variables are positive 25

D. Convex sets and concave functions 30

E. Sufficient conditions for a maximum 42

Presentation slides



A. Review: Differentiating a function of a function

1( ,..., )nx x x= an n-vector

jx ∈ 1,...,j n= , nx∈



Basics of maximization



jx ∈ 1,...,j n= , nx∈

1( ) ( ( ),..., ( ))nx t x t x t= where ( ), 1,...,jx t j n= is continuously differentiable.

1( ) ( ( ),..., ( ))ndxdxdx t t tdt dt dt

= the vector of derivatives of each component of ( )x t



Basics of maximization



jx ∈ 1,...,j n= , nx∈

1( ) ( ( ),..., ( ))nx t x t x t= where ( ), 1,...,jx t j n= is continuously differentiable.

1( ) ( ( ),..., ( ))ndxdxdx t t tdt dt dt

= the vector of derivatives of each component of ( )x t

The function ( )f x mapping from n-vectors to the real line

: nf →

( )f x a differentiable function (all partial derivatives are defined)



Function of a function: ( ) ( ( ))y t f x t=

n=2

1( ) ( ( )) ( ( ),..., ( ))ny t f x t f x t x t= =



Chain Rule: 1

( ) ( ( ))n

j

t j

dxdy ft x tdt x dt=

∂=

∂∑ .



Chain Rule: 1

( ) ( ( ))n

j

t j

dxdy ft x tdt x dt=

∂=

∂∑ .

Gradient vector: 1

( ) ( ( ),... ( ))n

f f fx x xx x x∂ ∂ ∂

≡∂ ∂ ∂

Vector notation: product of two n-vectors anda b 1

n

j jj

a b a b=

⋅ = ∑ .

Then ( ) ( ( ))dy f dxt x tdt x dt

∂= ⋅∂



Sketch of a proof (n=2)

1 2( ) ( ( ), ( ))y s f x s x s=

1 2( ) ( ( ), ( ))y s h f x s h x s h+ = + +




1 2( ) ( ( ), ( ))y s f x s x s=

1 2( ) ( ( ), ( ))y s h f x s h x s h+ = + +

( ) ( )y s h y s+ − 1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s h f x s x s= + + −

1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s f x s x s= + −

1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s h f x s h x s+ + + − +




1 2( ) ( ( ), ( ))y s f x s x s=

( ) ( )y s h y s+ −

1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s h f x s x s= + + −

1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s f x s x s= + −

1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s h f x s h x s+ + + − +

( ) ( )y s h y sh

+ −= 1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s f x s x s

h+ −

+ 1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s h f x s h x sh

+ + − +




1 2( ) ( ( ), ( ))y s f x s x s=

( ) ( )y s h y s+ −

1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s h f x s x s= + + −

1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s f x s x s= + −

1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s h f x s h x s+ + + − +

( ) ( )y s h y sh

+ −= 1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s f x s x s

h+ −

+ 1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s h f x s h x sh

+ + − +

= 1 2 1 2 1 1

1 1

( ( ), ( )) ( ( ), ( )) ( ) ( )[ ][ ]( ) ( )

f x s h x s f x s x s x s h x sx s h x s h

+ − + −+ −

+ 1 2 1 2 2 2

2 2

( ( ), ( )) ( ( ), ( )) ( ) ( )[ ][ ]( ) ( )

f x s h x s h f x s h x s x s h x sx s h x s h

+ + − + + −+ −




1 2( ) ( ( ), ( ))y s f x s x s=

( ) ( )y s h y s+ −

1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s h f x s x s= + + −

1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s f x s x s= + −

1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s h f x s h x s+ + + − +

( ) ( )y s h y sh

+ −= 1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s f x s x s

h+ −

+ 1 2 1 2( ( ), ( )) ( ( ), ( ))f x s h x s h f x s h x sh

+ + − +

= 1 2 1 2 1 1

1 1

( ( ), ( )) ( ( ), ( )) ( ) ( )[ ][ ]( ) ( )

f x s h x s f x s x s x s h x sx s h x s h

+ − + −+ −

+ 1 2 1 2 2 2

2 2

( ( ), ( )) ( ( ), ( )) ( ) ( )[ ][ ]( ) ( )

f x s h x s h f x s h x s x s h x sx s h x s h

+ + − + + −+ −

Taking limits: 1 2

1 2

( ) ( ( )) ( ( ))dx dxdy f fs x s x sdt x dt x dt

∂ ∂= +∂ ∂



B. Necessary conditions

1( ,..., )nx x x= a vector of decision variables where each component of x is a real number.





( jx ∈ , 1,...,j n= , equivalently nx∈ )






( )f x is a mapping from the set n onto the set

Assume that all the partial derivatives of ( )f x exist and are continuous






( )f x is a mapping from the set n onto the set


Maximization problem

{ ( ) | }n

xMax f x x∈






( )f x mapping from the set n onto the set


Maximization problem

{ ( ) | }n

xMax f x x∈

Focus on jx . Write the vector of all other components of x as

1 1 1( ,..., , ,..., )j j j nx x x x x− − +=

Then the function ( )f x can be written as follows:

( ) ( , )j jf x f x x−=



Depict graph of f

0( , )j jy f x x−=

jx

0 0( , ) 0j jj

fslope x xx −

∂= >∂

y

0jx



Depict graph of f

If 0 0 0( ) ( , ) 0j jj j

f fx x xx x −

∂ ∂≡ >

∂ ∂ for some j then for 0jδ > and sufficiently small

0 0 0 0( , ) ( , )j j j j jf x x f x xδ − −+ > so f cannot take on its maximum at 0x .

0( , )j jy f x x−=

jx

0 0( , ) 0j jj

fslope x xx −

∂= >∂

y

0jx



Depict graph of f

If 0 0 0( ) ( , ) 0j jj j

f fx x xx x −

∂ ∂≡ >



By an identical argument, f cannot take on its maximum at 0x if 0( ) 0j

f xx∂

<∂

for some j.

0( , )j jy f x x−=

jx

0 0( , ) 0j jj

fslope x xx −

∂= >∂

y

0jx



Depict graph of f

If 0 0 0( ) ( , ) 0j jj j

f fx x xx x −

∂ ∂≡ >



By an identical argument, f cannot take on its maximum at 0x if 0( ) 0j

f xx∂

<∂

for some j

Thus necessary conditions for f to take on its maximum at 0x are 0( ) 0, 1,...,j

f x j nx∂

= =∂

In vector notation, the gradient vector 0 0 0

1

( ) ( ( ),..., ( ))n

f f fx x xx x x∂ ∂ ∂

=∂ ∂ ∂

is the zero vector.

0( , )j jy f x x−=

jx

0 0( , ) 0j jj

fslope x xx −

∂= >∂

y

0jx



An alternative approach

Consider a weighted average xλ of any vectors 0 1, nx x ∈ where the weights are strictly positive and add to 1, that is 0 1(1 )x x xλ λ λ= − + where 0 1λ< < .





For any 0δ ≠ define 1 0x x δ= + .

Substituting for 1x , 0 0(1 ) ( )hx x x xλ λ λ δ λδ= − + + = +

Define the function of a function

0( ) ( ) ( )g f x f xλλ λδ= = + .








0( ) ( ) ( )g f x f xλλ λδ= = + .

Then

0

1( ) ( ) ( )

n

jj j

f fg x xx x

λλ λδ δ δ=

∂ ∂′ = + = ⋅∂ ∂∑








0( ) ( ) ( )g f x f xλλ λδ= = + .

Then

0

1( ) ( ) ( )

n

jj j

f fg x xx x

λλ λδ δ δ=

∂ ∂′ = + = ⋅∂ ∂∑

Setting 0λ =

0(0) ( )fg xx

δ∂′ = ⋅∂



Arguing as above, a necessary condition for f to take on its maximum at 0x is that ( )g λ has a maximum at 0λ = .

Therefore

0(0) ( ) 0fg xx

δ∂′ = ⋅ =∂

This must be true for all δ .

Choose (0,..., ,0,...,0)jδ δ= where 0jδ ≠ .

Then 0( ) 0j

f xx∂

=∂

It follows that if f takes on its maximum at 0x then

0( ) 0f xx∂

=∂

.



C. Maximization when the decision vector must be positive

0, 1,...,jx j n≥ =

( , 1,...,jx j n+∈ = , equivalently, nx +∈ )

Graph of f



C. Maximization when the decision vector must be positive

0, 1,...,jx j n≥ =

( , 1,...,jx j n+∈ = , equivalently, nx +∈ )

Graph of f

Necessary conditions for f to take on its maximum at 0x are as follows:

0( ) 0, 1,...,j

f x j nx∂

≤ =∂

with equality if 0 0jx >



Equivalently the gradient vector 0( )f xx∂∂

is negative and the inner product of 0x and the gradient

vector is the zero vector.

0 00, ( ) 0, 1,...,fx x j nx∂

≥ ≤ =∂

and 0 0( ) 0fx xx∂⋅ =∂

Since only one of the two inequality conditions can be strict, these conditions are often called the complementary slackness conditions.



D. Convex sets and concave functions

Definition: Convex combination

A convex combination xλ of vectors 0x and 1x is a weighted average

where the weights are strictly positive and add to 1.

0 1 0 1 0(1 ) ( )x x x x x xλ λ λ λ= − + = + − where 0 1λ< <








Definition: Convex set

The set X is convex if for any 0x and 1x in X every convex combination xλ is also in X








Definition: Convex set

The set X is convex if for any 0x and 1x in X every convex combination xλ is also in X

Definition: Concave function

The function f is concave if

for any 0 1,x x and convex

combination 0 1(1 )x x xλ λ λ= − +

0 1(1 ) ( ) ( ) ( )f x f x f xλλ λ− + ≤

!!!!!!!!

B



Proposition: The differentiable function f is concave if and only if for any 0 1,x x

1 0 0 1 0( ) ( ) ( ) ( )ff x f x x x xx∂

≤ + ⋅ −∂

Proof (only if):

Define 0 1 0 1 0( ) (1 ) ( )x x x x x xλ λ λ λ= − + = + − ,

Note that 1 0jj j

dxx x

dλ= − so 1 0dx x x

dλ= − .




1 0 0 1 0( ) ( ) ( ) ( )ff x f x x x xx∂

≤ + ⋅ −∂

Proof (only if):


Note that 1 0jj j

dxx x


dλ= − .

Define ( ) ( ( ))y f xλ λ= .

1 0( ) ( ( )) ( ( )) ( )dy f dx fx x x xd x d x

λ λ λλ λ

∂ ∂= ⋅ = ⋅ −∂ ∂

(**)




1 0 0 1 0( ) ( ) ( ) ( )ff x f x x x xx∂

≤ + ⋅ −∂

Proof (only if):


Note that 1 0jj j

dxx x


dλ= − .

Define ( ) ( ( ))y f xλ λ= .

1 0( ) ( ( )) ( ( )) ( )dy f dx fx x x xd x d x

λ λ λλ λ

∂ ∂= ⋅ = ⋅ −∂ ∂

(**)

( ) (0) ( ( )) ( (0))y y f x f xλ λ− = − .

Appealing to convexity, for all λ in the open interval (0,1)

0 1 0 1 0( ( )) (1 ) ( ) ( ) ( ) ( ( ) ( ))f x f x f x f x f x f xλ λ λ λ≥ − + = + −




1 0 0 1 0( ) ( ) ( ) ( )ff x f x x x xx∂

≤ + ⋅ −∂

Proof (only if):


Note that 1 0jj j

dxx x


dλ= − .

Define ( ) ( ( ))y f xλ λ= .

1 0( ) ( ( )) ( ( )) ( )dy f dx fx x x xd x d x

λ λ λλ λ

∂ ∂= ⋅ = ⋅ −∂ ∂

(**)

( ) (0) ( ( )) ( (0))y y f x f xλ λ− = − .

Appealing to convexity, for all λ in the open interval (0,1)

0 1 0 1 0( ( )) (1 ) ( ) ( ) ( ) ( ( ) ( ))f x f x f x f x f x f xλ λ λ λ≥ − + = + −

Therefore

1 0( ) (0) ( ( ) ( ))y y f x f xλ λ− ≥ − .



We have argued that

1 0( ) (0) ( ( ) ( ))y y f x f xλ λ− ≥ − .

Divide both sides by (0,1)λ∈ .

1 0( ) (0) ( ) ( )y y f x f xλλ−

≥ −



We have argued that

1 0( ) (0) ( ( ) ( ))y y f x f xλ λ− ≥ − .


1 0( ) (0) ( ) ( )y y f x f xλλ−

≥ −

Take the limit as 0λ →

1 0(0) ( ) ( )dy f x f xdt

≥ −



We have argued that

1 0( ) (0) ( ( ) ( ))y y f x f xλ λ− ≥ − .


1 0( ) (0) ( ) ( )y y f x f xλλ−

≥ −


1 0(0) ( ) ( )dy f x f xdt

≥ −

We already established

1 0( ) ( ( )) ( ( )) ( )dy f dx fx x x xd x d x

λ λ λλ λ

∂ ∂= ⋅ = ⋅ −∂ ∂

(**)



We have argued that

1 0( ) (0) ( ( ) ( ))y y f x f xλ λ− ≥ − .


1 0( ) (0) ( ) ( )y y f x f xλλ−

≥ −


1 0(0) ( ) ( )dy f x f xdt

≥ −

We already established

1 0( ) ( ( )) ( ( )) ( )dy f dx fx x x xd x d x

λ λ λλ λ

∂ ∂= ⋅ = ⋅ −∂ ∂

(**)

Setting 0λ = and noting that 0(0)x x= it follows that

0 1 0 1 0( ) ( ) ( ) ( )f x x x f x f xx∂

⋅ − ≥ −∂



Proof: (if)

For any ,y z

( ) ( ) ( ) ( )ff y f z z y zx∂

≤ + ⋅ −∂

(i) Set z xλ= and 0y x= Then 0 0( ) ( ) ( ) ( )ff x f x x x xx

λ λ λ∂≤ + ⋅ −

∂

(ii) Set z xλ= and 1y x= Then 1 1( ) ( ) ( ) ( )ff x f x x x xx

λ λ λ∂≤ + ⋅ −

∂

Class exercise

Complete the proof.



E: Sufficient conditions for a maximum

Sufficient conditions for a maximum (with nx +∈ )

Suppose that f is concave.

If 0( ) 0, 1,...,f x j nx∂

≤ =∂

and 0 0( ) 0fx xx∂⋅ =∂

then 0x solves { ( ) | }n

xMax f x x +∈





If 0( ) 0, 1,...,f x j nx∂

≤ =∂

and 0 0( ) 0fx xx∂⋅ =∂


xMax f x x +∈

Proof: Since f is concave, for any 1 0x x≠

1 0 0 1 0 0 0 1 0

1( ) ( ) ( ) ( ) ( ) ( )( )

n

j jj j

f ff x f x x x x f x x x xx x=

∂ ∂≤ + ⋅ − = + −

∂ ∂∑





If 0( ) 0, 1,...,f x j nx∂

≤ =∂

and 0 0( ) 0fx xx∂⋅ =∂


xMax f x x +∈

Proof: Since f is concave, for any 1 0x x≠

1 0 0 1 0 0 0 1 0

1( ) ( ) ( ) ( ) ( ) ( )( )

n

j jj j

f ff x f x x x x f x x x xx x=

∂ ∂≤ + ⋅ − = + −

∂ ∂∑

Either (i) 0( ) 0j

f xx∂

=∂

or (ii) 0( ) 0j

f xx∂

<∂

and 0 0jx = in which case 1 1 0 0j j jx x x= − ≥ .

Thus the summation is negative and so 1 0( ) ( )f x f x≤

Basics of maximization - Essential...

Documents

Transcript of Basics of maximization - Essential...