V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples,...

1

V. Linear & Nonlinear Least-Squares

V.0.Examples, linear/nonlinear least-squares

V.1.Linear least-squares

V.1.1.Normal equations

V.1.2.The orthogonal transformation method

V.2.Nonlinear least-squares

V.2.1.Newton method

V.2.2.Gauss-Newton method


2

V.0. Examples, linear/nonlinear least-squares

In practice, one has often to determine unknown parameters of agiven function (from natural laws or model assumptions) through aseries of measurements.

Usually the number of measurements m is much bigger than thenumber of parameters n, i.e.

Measurement

Time

Quantity

Model: Goal: Find1

2


3


In practice, one has often to determine unknown parameters of agiven function (from natural laws or model assumptions) through aseries of measurements.

Usually the number of measurements m is much bigger than thenumber of parameters n, i.e.

Measurement

Time

Quantity

Model: Goal: Find1

2

Problem: more equations than unknowns! … overdetermined


4


Goal: Find1

Data:

Linear in model parameters!

Model:


5


1

Data:

JUSTNOTATION!

Goal: Find


Model:


6


● Idea: choose the parameters such that thedistance between the data and the curve isminimal, i.e. the curve that fits best.


7



Model: 1


8



MINIMIZE!


9



● Least squares solution:

Euclidean distance,a.k.a. 2-normDomain of valid parameters

(from application!)


10


Model: Goal: Find2

Data:

Nonlinear in model parameters!

JUSTNOTATION!


11




12



MINIMIZE!


13


● General problem:

Lin

ear

No

nlin

ear

In parameters!

measurements parameters

Overdetermined!

usually...


14



● Define scalar-valued function


Linear

Nonlinear

Linear

Nonlinear


15


● Quiz: linear or nonlinear model?

1)

2) Model parameters:

Model parameters:


16


● Quiz: linear or nonlinear model?

1)

2) Model parameters:

Model parameters:


17


● General problem: measurements parameters

Overdetermined!

Least squares solution:

Assumption: has full column rank (linearly indep.)


18


● Rewrite: Transpose!

Scalar!

~

~

V.1.1. Normal equations


19

● Gradient of

must vanish at extremum (min/max):

Gradient

(Gauss') Normal equations

Nothing difficult…

~



20

● Necessary condition:

● Is it a minimum?We have to make sure that the matrix ispositive definite

(Gauss') Normal equations

1

2

Norm (length!)

Unless Because of our assumptionof rank nColumns linearly independent!

Not sufficient!

~



21

● Geometric interpretation of the normalequations

● It turns out that they lead to a worserconditioned problem, i.e. difficult to solve thenormal equations numerically…Therefore the next method is preferred

.



22

V.1.2. The orthogonal decomposition method

● Definition:

● Key property:

An orthogonal matrix is a real squarematrix whose columns and rows are orthogonalunit vectors

This means:

Orthogonal matrices leave the Euclidean lengthinvariant


23

Every matrix with full column rank(the columns are linearly independent) has a so-calledQR-decomposition

where is an orthogonal matrix and an uppertriangular matrix

!Non zero diagonal!

Matlab: [Q,R]=qr(A)

Fact:



24

Every matrix with full column rank(the columns are linearly independent) has a so-calledQR-decomposition

where is an orthogonal matrix and an uppertriangular matrix

Matlab: [Q,R]=qr(A)

Fact:

Upper triangular



25

Minimize residuum:

Orthogonal!We still solve the sameminimization problem!!!



26

Minimize residuum:

Orthogonal!We still solve the sameminimization problem!!!



27

V.1. Example: linear least-squares

Goal: Find1

Data:


Model:


28

V.1. Example: linear least-squares

Goal: Find1

Data:

Model:

Normal equations:

Orthogonal transformation: analogue…


29

V.2. Nonlinear least-squares

usually...

● General problem: measurements parameters

Overdetermined!

Least squares solution:


30

● Gradient of

must vanish at extremum (min/max):

Gradient

V.2.1 Newton method

equations

unknowns!


31

V.2.1 Newton method

Hessian

Gradient

Apply Newton's method to solve

NOT


32

Linearize the residuum/error equations

V.2.2 Gauss-Newton method

Linearize at some

Linear least squares problem!


33

V.2.2 Gauss-Newton method

Given an initial guess:

Solve a sequence of linear least squares problems

Until convergence

Small enough

Problem:

...


34

V.2. Example: nonlinear least-squares

Model: Goal: Find2

Data:

Nonlinear in model parameters!


35


Model: Goal: Find2

Data:

Newton method:


36


Model: Goal: Find2

Data:

Newton method:

Gradient

Two equations in two unknowns!


37


Model: Goal: Find2

Data:

Newton method:

Hessian


38


Model: Goal: Find2

Data:

Newton method:

Iterate!

Initial guess:


39


Model: Goal: Find2

Data:

Gauss-Newton method:

Linearize residuum/error equations:

Linear in parameters now!


40

V. Summary

● Linear/Nonlinear in parameters

● Overdetermined system of equations!Determine parameters in the least-squares sense...

● Linear:

– Normal equations

– Orthogonal transformation method

– Example

● Nonlinear:

– Newton method

– Gauss-Newton method

– Example


V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples,...

Documents

Transcript of V. Linear & Nonlinear Least-Squareskaroger/numci/2016/files/Chap05/...3 V.0. Examples,...