ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept....
-
Upload
dylan-wood -
Category
Documents
-
view
218 -
download
0
description
Transcript of ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept....
![Page 1: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/1.jpg)
ABE425 Engineering Measurement Systems
Ordinary Least Squares (OLS) Fitting
Dr. Tony E. Grift
Dept. of Agricultural & Biological EngineeringUniversity of Illinois
![Page 2: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/2.jpg)
First, we need to talk about linearity
In Mathematics, there are linear and non-linear operations:
If an operation is linear, the superposition principle can be applied:
a b
a b
Operation
Operation
1 1
2 2
a a b bOperation1 2 1 2
k a a k b bOperation1 2 1 2
![Page 3: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/3.jpg)
Examples
Multiplication by a constant c (linear)
a a c
a a c
Operation
Operation
1 1
2 2
a a c a c a c a a cOperation1 2 1 2 1 2
![Page 4: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/4.jpg)
Examples
Differentiation (linear)
a a c
a a c
Operation
Operation
1 1
2 2
d f x g x d f x d g xdx dx dx
![Page 5: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/5.jpg)
Examples
Integration (linear)
f x g x dx f x dx g x dx
![Page 6: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/6.jpg)
Examples
Squaring (non-linear)
a a
a a
Operation
Operation
1 12
2 22
a a a a a aOperation1 2
2
12
22
1 2
2
![Page 7: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/7.jpg)
Examples
Square root (non-linear)
a a
a a
Operation
Operation
1 1
2 2
a a a a a aOperation1 2 1 2 1 2
![Page 8: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/8.jpg)
Try these:
sin
ln
xe
x
x
![Page 9: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/9.jpg)
You collected a set of data pairs (for example temperature versus time
>> x=[0:1:10]'>> y = [0.5 0.75 1.25 1.3 2.1 2.0 3.1 3.05 4.0 4.5 5]'
, , 1,i ix y i m
0 1 2 3 4 5 6 7 8 9 100.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Time (s)
Tem
pera
ture
(deg
C)
![Page 10: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/10.jpg)
Model is some function of the independent variable and the parameter vector. You choose the model!!
The error is the difference between a data point and the corresponding model
,iy f x
,i i ie y f x
Error = Data - Model
![Page 11: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/11.jpg)
The idea of using the sum of least squared errors came from Gauss (first published by Legendre):
How can we minimize this error with respect to the parameter vector? In other words, how can we find the parameter vector that minimizes the error and maximizes the fit?
2
1
i m
ii
S e
“Sur la Méthode des moindres quarrés” in Legendre’s Nouvellesméthodes pour la détermination des orbites des comètes, Paris 1805.
![Page 12: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/12.jpg)
The minimum value of the sum of squares is obtained by setting this partial derivative to zero
The derivative is partial, because the Sum of residuals S is a function of the error and the error itself is a function of the parameter vector (remember the chain rule):
* i
j i j
eS Se
2
1 1
2i m i m
i ii ii
SS e ee
,, ii
i i ij j
f xee y f x
S e
![Page 13: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/13.jpg)
The minimum value of the sum of squares is obtained by setting this partial derivative to zero
Substitution of the results from the previous slide gives:
Now, we need to find out what is
1
,2i m
ii
ij j
f xS e
,ij
f x
![Page 14: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/14.jpg)
The proposed model is linear in the parameters. Here is an polynomial example:
For each ith measurement this can be written using a matrix and a parameter vector as follows:
0 1 2 30 1 2 3 ..f x x x x x
This can also be written as:
0
10 1 2 3
2
3
,i i i i if x x x x x
1
,j n
i ij jj
f x A
![Page 15: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/15.jpg)
For all measurement points we obtain:
3 2 1 01
3
2
1
03 2 1 0
| | | |, | | | |
| | | |
i i i i
m m m m m
A
x x x x x
f
x x x x x
This can also be written in vector form as:
Where A is the regressor matrix
,f x A
![Page 16: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/16.jpg)
From the model definition we can obtain the partial derivative with respect to the parameter vector
By replacing the error with the (data – model) we get :
1
,,
j ni
i ij j ijj j
f xf x A A
1 1
,2 2i m i m
ii i ij
i ij j
f xS e e A
1
,j n
i i i i ij jj
e y f x y A
![Page 17: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/17.jpg)
Upon rearrangement these become n simultaneous linear equations, the normal equations.
1 1
2 0i m k n
i ik k iji kj
S y A A
1 1 1
, 1,i m k n i m
ij ik k ij ii k i
A A A y j n
T TA A A y
1ˆ T TA A A y
![Page 18: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/18.jpg)
Here a second order polynomial with intercept was applied
0 1 2 3 4 5 6 7 8 9 100
1
2
3
4
5
6
Time (s)
Tem
pera
ture
(deg
C)
2 1 02 1 0f x x x x
![Page 19: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/19.jpg)
Determine whether the data needs an intercept. Often physical constraints demand that the fit curve passes through the origin!
2 12 1f x x x
0 1 2 3 4 5 6 7 8 9 100
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
No intercept!
![Page 20: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/20.jpg)
OLS labfunction [theta, msq] = fitols(x,y,Ovec)% Fit polynomial function on data in OLS sense% Author :% Date :% Revision :%% Syntax : [theta,msq] = fitols(x,y,Ovec)%% theta : Parameter vector% msq : Mean square error% x : Independent variable% y : Dependent variable%% Ovec indicates terms [1 x x^2 ..]*Ovec'% Example Ovec = [1 0 1] gives [1 x^2] and not x % If vectors x,y are horizontal, transpose them to make them vertical % Make sure the x and y vector have the same length. If not alert the user% with an error dialog box (type help errordlg ) % Build the matrix of regressors. Check each entry of Ovec, and if it is a% 1, add another column to the regression matrix A. A = []; % Compute the parameter vector theta using the OLS formula % Compute the error vector % Compute the mean square error which indicates how good the fit is % Plot y (Temperature in C) versus x (Current in A). Add labels and title. % Your output should look as shown in the handout.
![Page 21: ABE425 Engineering Measurement Systems Ordinary Least Squares (OLS) Fitting Dr. Tony E. Grift Dept. of Agricultural & Biological Engineering University.](https://reader033.fdocuments.in/reader033/viewer/2022052707/5a4d1b417f8b9ab0599a0f8c/html5/thumbnails/21.jpg)
ABE425 Engineering Measurement Systems
Ordinary Least Squares (OLS) Fitting
The End
Dept. of Agricultural & Biological EngineeringUniversity of Illinois