Review of elements of Calculus(functions in more than one variable)

Partly adapted from the lectures of prof Piero Fariselli(University of Bologna)

yx

z 2 2, 100z f x y x y

x

y

10

10

10

10

100

sketch of graph level curves

Level curves are drawn by holding the z value constant (similar to contour lines on a topographic map.)

Functions of two variables

Partial derivatives are defined as derivatives of a function of multiple variables when all but the variable of interest are held fixed during the differentiation.

Definition of Partial Derivatives of a Function of Two VariablesIf z = f(x,y), the the first partial derivatives of f with respect to x and y are the functions fx and fy defined by

0

0

, ( , ), lim

, ( , ), lim

x x

y y

f x x y f x yf x y

x

f x y y f x yf x y

y

Provided the limits exist.

Partial Derivatives

To find the partial derivatives, hold one variable constant and differentiate with respect to the other.

Example 1: Find the partial derivatives fx and fy for the function

4 2 2 3( , ) 5 2f x y x x y x y

To find the partial derivatives, hold one variable constant and differentiate with respect to the other.

Example 1: Find the partial derivatives fx and fy for the function

4 2 2 3( , ) 5 2f x y x x y x y

Solution:

4 2 2 3

3 2 2

2 3

( , ) 5 2

( , ) 20 2 6

( , ) 2 2

x

y

f x y x x y x y

f x y x y x yx

f x y x y x

Notation for First Partial Derivative

For z = f(x,y), the partial derivatives fx and fy are denoted by

( , ) ,

( , ) ,

x x

y y

zf x y f x y z

x x

and

zf x y f x y z

y y

The first partials evaluated at the point (a,b) are denoted by

( , ) ( , ), ,a b x a b y

z zf a b and f a b

x y

Example 2: Find the partials fx and fy and evaluate them at the indicated point for the function

( , ) (2, 2)xy

f x y atx y

Example 2: Find the partials fx and fy and evaluate them at the indicated point for the function

( , ) (2, 2)xy

f x y atx y

Solution:

2 2

2 2 2

2

2

2 2

2 2 2

2

2

( , ) (2, 2)

,( ) ( ) ( )

2 4 12, 2

16 4(2 2 )

,( ) ( ) ( )

4 12, 2

16 4( )

x

x

y

y

xyf x y at

x y

x y y xy xy y xy yf x y

x y x y x y

f

x y x xy x xy xy xf x y

x y x y x y

xf

x y

The following slide shows the geometric interpretation of the partial derivative. For a fixed x, z = f(x0,y) represents the curve formed by intersecting the surface z = f(x,y) with the plane x = x0.

0 0,xf x y represents the slope of this curve at the point (x0,y0,f(x0,y0))

Thanks to http://astro.temple.edu/~dhill001/partial-demo/For the animation.

Definition of Partial Derivatives of a Function of Three or More VariablesIf w = f(x,y,z), then there are three partial derivatives each of which is formed by holding two of the variables

0

0

0

, , ( , , ), , lim

, , ( , , ), , lim

, , ( , , ), , lim

x x

y y

z z

f x x y z f x y zwf x y z

x x

f x y y z f x y zwf x y z

y y

f x y z z f x y zwf x y z

z z

In general, if

1 2

1 2

( , ,... )

, ,... , 1,2,...k

n

x nk

w f x x x there are n partial derivatives

wf x x x k n

x

where all but the kth variable is held constant

Notation for Higher Order Partial Derivatives

Below are the different 2nd order partial derivatives:

yx

xy

yy

xx

fyx

f

y

f

y

fxy

f

x

f

y

fy

f

y

f

y

fx

f

x

f

x

2

2

2

2

2

2

Differentiate twice with respect to x

Differentiate twice with respect to y

Differentiate first with respect to x and then with respect to y

Differentiate first with respect to y and then with respect to x

Theorem

If f is a function of x and y such that fxy and fyx are continuous on an open disk R, then, for every (x,y) in R,

fxy(x,y)= fyx(x,y)

Example 3:

Find all of the second partial derivatives of yxyxyyxf 22 523),(

Work the problem first then check.

Example 3:

Find all of the second partial derivatives of yxyxyyxf 22 523),(

xyyxf

xxyyxf

yxyxyyxf

xyyxf

xyyyxf

yxyxyyxf

xyxf

xxyyxf

yxyxyyxf

yyxf

xyyyxf

yxyxyyxf

yx

y

xy

x

yy

y

xx

x

106),(

526),(

523),(

106),(

103),(

523),(

6),(

526),(

523),(

10),(

103),(

523),(

2

22

2

22

2

22

2

22

Notice that fxy = fyx

Example 4: Find the following partial derivatives for the function

zxyezyxf x ln),,(

a. xzf

b. zxf

c. xzzf

d. zxzf

e. zzxf

Work it out then go to the next slide.

Example 4: Find the following partial derivatives for the function

zxyezyxf x ln),,(

a. xzf

b. zxf

zzyxf

zyezyxf

zxyezyxf

xz

x

x

x

1),,(

ln),,(

ln),,(

zzyxf

z

xzyxf

zxyezyxf

zx

z

x

1),,(

),,(

ln),,(

Again, notice that the 2nd

partials fxz = fzx

c. xzzf

d. zxzf

e. zzxf

2

1),,(

1),,(

ln),,(

ln),,(

zzyxf

zzyxf

zyezyxf

zxyezyxf

xzz

xz

x

x

x

2

1),,(

1),,(

),,(

ln),,(

zzyxf

zzyxf

z

xzyxf

zxyezyxf

zxz

zx

z

x

2

2

1),,(

),,(

),,(

ln),,(

zzyxf

z

xzyxf

z

xzyxf

zxyezyxf

zzx

zz

z

x

NoticeAll

Are Equal

f: RnR. If f(x) is of class C2, objective function

Gradient of f

Is a vector containing all the partial derivatives of the first order

Gradient

Given a function f(xy) and a level curve f(x,y) = c

the gradient of f is:

Consider 2 points of the curve: (x,y); (x+εx, x+εy), for small ε

The Gradient is locally perpendicular to level curves

y

f

x

ff ,

(x,y)(x+εx, y+εy)

),(

),(),(

,

,,

yx

T

yx

y

yx

xyx

gyxf

y

f

x

fyxfyxf

ε

Since both : (x,y); (x+εx, x+εy), points satisfy the curve equation:

The gradient is perpendicular to ε.For small ε, ε is parallel to the curve and,by consequence, thegradient is perpendicular to the curve.

The gradient points towards the direction of maximumincrease of f

The local perpendicular to a curve: Gradient

(x,y)(x+εx, x+εy)

0),(

),(

yx

T

yx

T

f

fcc

ε

ε

ε

grad (f)

f: RnR. If f(x) is of class C2, objective function

Hessian of f

Is a square matrix of order n containing all the partial derivatives of the second order

H=

Since the mixed partial derivatives are equal irrespectively of the order of derivation, the matrix is symmetric

Hessian

Taylor expansion

If the variables are represented with a n-valued vector x, the Taylor expansion around a point x* is

.....*)(*)(2

1*)(*)()(

* xxHxxxxxfxfxf T

x

Local optima (without constraints)

The condition to have a local optimum in x* is

i.e.: all the partial derivatives must be null

In order to study whether the point is a minimum or a maximum, the matrix H must be considered

H is a nxn symmetric real matrix: it has n real eigenvalues

The point x* isa maximum if all eigenvectors are positive, a minimum if all the eigenvectors are negative

)(

*0 n

xf

Saddle points

When eigenvalues of the Hessian matrix have differentsign , the critical poin is a Saddle point

Constrained optimization problems: Lagrange Multipliers

10/12/2015 28

We want to maximise the function z = f(x,y)subject to the constraint g(x,y) = c (curve in the x,y plane)

Aim

Solve the constraint g(x,y) = c and express, for example, y=h(x)

The substitute in function f and find the maximum in x of

f(x, h(x))

Analytical solution of the constraint can be very difficult

Simple solution

Contour lines of a function

Contour lines of f and constraint

Suppose we walk along the constraint line g (x,y)= c.

In general the contour lines of f are distinct from the constraint g (x,y)= c.

While moving along the constraint line g (x,y)= c the value of fvary (that is, different contour levels for f are intersected).

Only when the constraint line g (x,y)= c touches the contour lines of f in a tangential way, we do not increase or decrease the value of f: the function f is at its local max or min along the constraint.

Lagrange Multipliers

Geometrical interpretation

Contour line and constraint are tangential: their localperpendicular to are parallel

Normal to a curve

Lagrange Multipliers

On the point of g(x,y)=c that

Max-min-imize f(x,y), the gradient

of f is perpendicular to the curve

g(x,y) =c, otherwise we should increase or decrease f by

moving locally on the curve.

So, the two gradients are parallel

for some scalar λ (where is the gradient).

Thus we want points (x,y) where g(x,y) = c and

,

To incorporate these conditions into one equation, we introduce

an auxiliary function (Lagrangian)

and solve

.

Lagrange Multipliers

cyxgyxfyxF ),(),(),,(

Recap of Constrained Optimization

Suppose we want to: minimize/maximize f(x) subject to g(x) = 0

A necessary condition for x0 to be a solution:

a: the Lagrange multiplier

For multiple constraints gi(x) = 0, i=1, …, m, we need a Lagrange multiplier ai for each of the constraints

-

-