Steepest Decent and Conjugate Gradients (CG)

Steepest Decent and Conjugate Gradients (CG)

• Solving of the linear equation system bAx

Steepest Decent and Conjugate Gradients (CG)

• Solving of the linear equation system

• Problem: dimension n too big, or not enough time for gauss elimination

Iterative methods are used to get an approximate solution.

bAx

Steepest Decent and Conjugate Gradients (CG)

• Solving of the linear equation system

• Problem: dimension n too big, or not enough time for gauss elimination

Iterative methods are used to get an approximate solution.

• Definition Iterative method: given starting point , do steps

hopefully converge to the right solution

bAx

0x,, 21 xx

x

starting issues

starting issues

• Solving is equivalent to minimizing bAx cxbAxxxf TT

2

1:)(

starting issues

• Solving is equivalent to minimizing

• A has to be symmetric positive definite:

bAx cxbAxxxf TT

2

1:)(

00 xAxxAA TT

starting issues

• 02

1

2

1)(

!

bAxbAxxAxfsymmetricA

T

starting issues

•

• If A is also positive definite the solution of is the minimum

02

1

2

1)(

!

bAxbAxxAxfsymmetricA

T

bAx

starting issues

•

• If A is also positive definite the solution of is the minimum

02

1

2

1)(

!

bAxbAxxAxfsymmetricA

T

bAx

00

11

2

1

2

1)(

d

TT AddcbAbdbAf

starting issues

• error:

The norm of the error shows how far we are away from the exact solution, but can’t be computed without knowing of the exact solution .

xxe ii :

x

starting issues

• error:

The norm of the error shows how far we are away from the exact solution, but can’t be computed without knowing of the exact solution .

• residual:

can be calculated

xxe ii :

x)(: xfAeAxbr iii

Steepest Decent

Steepest Decent

• We are at the point . How do we reach ?ix 1ix

Steepest Decent

• We are at the point . How do we reach ?

• Idea: go into the direction in which decreases most quickly ( )

ix 1ix

)(xf

ii rxf )(

Steepest Decent

• We are at the point . How do we reach ?

• Idea: go into the direction in which decreases most quickly ( )

• how far should we go?

ix 1ix

)(xf

ii rxf )(

Steepest Decent

• We are at the point . How do we reach ?

• Idea: go into the direction in which decreases most quickly ( )

• how far should we go?

Choose so that is minimized:

ix 1ix

)(xf

ii rxf )(

)( ii rxf

Steepest Decent

• We are at the point . How do we reach ?

• Idea: go into the direction in which decreases most quickly ( )

• how far should we go?

Choose so that is minimized:

ix 1ix

)(xf

ii rxf )(

)( ii rxf

0)( ii rxfd

d

Steepest Decent

• We are at the point . How do we reach ?

• Idea: go into the direction in which decreases most quickly ( )

• how far should we go?

Choose so that is minimized:

ix 1ix

)(xf

ii rxf )(

)( ii rxf

0)( ii rxfd

d

0)( iT

ii rrxf

Steepest Decent

• We are at the point . How do we reach ?

• Idea: go into the direction in which decreases most quickly ( )

• how far should we go?

Choose so that is minimized:

ix 1ix

)(xf

ii rxf )(

)( ii rxf

0)( ii rxfd

d

0)( iT

ii rrxf

0))(( iT

ii rbrxA

Steepest Decent

• We are at the point . How do we reach ?

• Idea: go into the direction in which decreases most quickly ( )

• how far should we go?

Choose so that is minimized:

ix 1ix

)(xf

ii rxf )(

)( ii rxf

0)( ii rxfd

d

0)( iT

ii rrxf

0))(( iT

ii rbrxA

iT

r

iiT

i rAxbrAr

i

)()(

Steepest Decent

• We are at the point . How do we reach ?

• Idea: go into the direction in which decreases most quickly ( )

• how far should we go?

Choose so that is minimized:

ix 1ix

)(xf

ii rxf )(

)( ii rxf

0)( ii rxfd

d

0)( iT

ii rrxf

0))(( iT

ii rbrxA

iT

r

iiT

i rAxbrAr

i

)()( i

Ti

iTi

Arr

rr

Steepest Decent

one step of steepest decent can be calculated as follows:

iiii

Ti

iTi

i

ii

rxx

Arr

rr

Axbr

1

Steepest Decent

one step of steepest decent can be calculated as follows:

• stopping criterion: or with an given small

It would be better to use the error instead of the residual, but you can’t calculate the error.

iiii

Ti

iTi

i

ii

rxx

Arr

rr

Axbr

1

maxii 0rri

Steepest Decent

Method of steepest decent:

1

)(

0

00max

0

0

ii

Axbr

rxxArr

rr

rrrrandiiwhile

rr

Axbr

i

T

T

TT

Steepest Decent

• As you can see the starting point is important!

Steepest Decent

• As you can see the starting point is important!

When you know anything about the solution use it to guess a good starting point. Otherwise you can choose a starting point you want e.g. .00 x

Steepest Decent - Convergence

Steepest Decent - Convergence

• Definition energy norm: Axxx T

A:

Steepest Decent - Convergence

• Definition energy norm:

• Definition condition:

( is the largest and the smallest eigenvalue of A)

Axxx T

A:

min

max:

max min

Steepest Decent - Convergence

• Definition energy norm:

• Definition condition:

( is the largest and the smallest eigenvalue of A)

•

convergence gets worse when the condition gets larger

Axxx T

A:

min

max:

max min

A

i

Aiee 01

1

Conjugate Gradients

Conjugate Gradients

• is there a better direction?

Conjugate Gradients

• is there a better direction?

• Idea: orthogonal search directions110 ,,, nddd

Conjugate Gradients

• is there a better direction?

• Idea: orthogonal search directions110 ,,, nddd

1

0

n

iiidx

Conjugate Gradients

• is there a better direction?

• Idea: orthogonal search directions

• only walk once in each direction and minimize

110 ,,, nddd

1

0

n

iiidx

Conjugate Gradients

• is there a better direction?

• Idea: orthogonal search directions

• only walk once in each direction and minimize

maximal n steps are needed to reach the exact solution

110 ,,, nddd

1

0

n

iiidx

Conjugate Gradients

• is there a better direction?

• Idea: orthogonal search directions

• only walk once in each direction and minimize

maximal n steps are needed to reach the exact solution

has to be orthogonal to

110 ,,, nddd

1

0

n

iiidx

1 ie id

Conjugate Gradients

• example with the coordinate axes as orthogonal search directions:

Conjugate Gradients

• example with the coordinate axes as orthogonal search directions:

Problem: can’t be computed

because

(you don’t know !)

iTi

iTi

idd

ed

ie

Conjugate Gradients

• new idea: A-orthogonal110 ,,, nddd

Conjugate Gradients

• new idea: A-orthogonal

• Definition A-orthogonal: A-orthogonal

(reminder: orthogonal: )

110 ,,, nddd

ji dd , 0 jTi Add

ji dd , 0 jTi dd

Conjugate Gradients

• new idea: A-orthogonal

• Definition A-orthogonal: A-orthogonal

(reminder: orthogonal: )

• now has to be A-orthogonal to

110 ,,, nddd

ji dd , 0 jTi Add

ji dd , 0 jTi dd

1ie id

Conjugate Gradients

• new idea: A-orthogonal

• Definition A-orthogonal: A-orthogonal

(reminder: orthogonal: )

• now has to be A-orthogonal to

110 ,,, nddd

ji dd , 0 jTi Add

ji dd , 0 jTi dd

1ie id

iTi

iTi

iTi

iTi

i Add

rd

Add

Aed

Conjugate Gradients

• new idea: A-orthogonal

• Definition A-orthogonal: A-orthogonal

(reminder: orthogonal: )

• now has to be A-orthogonal to

can be computed!

110 ,,, nddd

ji dd , 0 jTi Add

ji dd , 0 jTi dd

1ie id

iTi

iTi

iTi

iTi

i Add

rd

Add

Aed

Conjugate Gradients

• A set of A-orthogonal directions can be found with n linearly independent vectors and conjugate Gram-Schmidt (same idea as Gram-Schmidt).

iu

Conjugate Gradients

• Gram-Schmidt:

linearly independent vectors10 ,, nuu

Conjugate Gradients

• Gram-Schmidt:

linearly independent vectors10 ,, nuu

jTj

jTi

ij

i

jjijii

dd

du

dudi

ud

1

0

00

:0

Conjugate Gradients

• Gram-Schmidt:

linearly independent vectors

• conjugate Gram-Schmidt:

10 ,, nuu

jTj

jTi

ij Add

Adu

jTj

jTi

ij

i

jjijii

dd

du

dudi

ud

1

0

00

:0

Conjugate Gradients

• A set of A-orthogonal directions can be found with n linearly independent vectors and conjugate Gram-Schmidt (same idea as Gram-Schmidt).

• CG works by setting (makes conjugate Gram-Schmidt easy)

iu

ii ru

Conjugate Gradients

• A set of A-orthogonal directions can be found with n linearly independent vectors and conjugate Gram-Schmidt (same idea as Gram-Schmidt).

• CG works by setting (makes conjugate Gram-Schmidt easy)

with1 iiii drd 11

i

Ti

iTi

i rr

rr

ii ru

iu

Conjugate Gradients

• 0:1

0

1

n

jkk

Tik

n

jkkk

Tii

Tij

Ti AdddAdAedrdji

Conjugate Gradients

•

•

0:1

0

1

n

jkk

Tik

n

jkkk

Tii

Tij

Ti AdddAdAedrdji

1

0

i

kkikii dud

Conjugate Gradients

•

•

0:1

0

1

n

jkk

Tik

n

jkkk

Tii

Tij

Ti AdddAdAedrdji

1

0

i

kkikii dud

1

00

0:i

kjk

jTkikj

Tij

Ti rdrurdji

Conjugate Gradients

•

•

0:1

0

1

n

jkk

Tik

n

jkkk

Tii

Tij

Ti AdddAdAedrdji

1

0

i

kkikii dud

1

00

0:i

kjk

jTkikj

Tij

Ti rdrurdji

jiru jTi 0

Conjugate Gradients

•

•

0:1

0

1

n

jkk

Tik

n

jkkk

Tii

Tij

Ti AdddAdAedrdji

1

0

i

kkikii dud

1

00

0:i

kjk

jTkikj

Tij

Ti rdrurdji

jiru jTi 0

ijjTi

jTiii

rr

jirrru

0:

Conjugate Gradients

•

•

•

0:1

0

1

n

jkk

Tik

n

jkkk

Tii

Tij

Ti AdddAdAedrdji

ijjTi

jTiii

rr

jirrru

0:

iTi

i

kjk

jTkiki

Tii

Ti rurdrurd

1

00

Conjugate Gradients

• jiAdd

Adr

jTj

jTi

ij

Conjugate Gradients

•

•

jiAdd

Adr

jTj

jTi

ij

jjjjjjjj AdrdeAAer )(11

Conjugate Gradients

•

•

jiAdd

Adr

jTj

jTi

ij

jjjjjjjj AdrdeAAer )(11

jTijj

Tij

Ti Adrrrrr 1

Conjugate Gradients

•

•

jiAdd

Adr

jTj

jTi

ij

jjjjjjjj AdrdeAAer )(11

jTijj

Tij

Ti Adrrrrr 1

1 jTij

Tij

Tij rrrrAdr

Conjugate Gradients

1 jTij

Tij

Tij rrrrAdr

10

11

jiji

jirr

jirr

Adr

rr

i

iTi

i

iTi

jTi

ijjTi

Conjugate Gradients

10 ji

ij

Conjugate Gradients

1

10

1111111

jirr

rr

rd

rr

Add

rrji

iTi

iTi

iTi

iTi

def

iTii

iTiij

Method of Conjugate Gradients:

00

0

i

rr

rd

Axbr

1

)( 00max

ii

drd

rr

rr

Axbr

rr

dxxAdd

rr

rrrrandiiwhile

oldTold

T

old

T

T

TT

Conjugate Gradients - Convergence

Conjugate Gradients - Convergence

• A

i

Aiee 0

1

12

Conjugate Gradients - Convergence

•

• for steepest decent for CG

Convergence of CG is much better!

A

i

Aiee 0

1

12