Least Squares Approximation: A Linear Algebra Technique
description
Transcript of Least Squares Approximation: A Linear Algebra Technique
![Page 1: Least Squares Approximation: A Linear Algebra Technique](https://reader036.fdocuments.in/reader036/viewer/2022062408/56813af5550346895da37947/html5/thumbnails/1.jpg)
Least Squares Approximation: A Linear Algebra Technique
By Larry Wong and James Sfregola
![Page 2: Least Squares Approximation: A Linear Algebra Technique](https://reader036.fdocuments.in/reader036/viewer/2022062408/56813af5550346895da37947/html5/thumbnails/2.jpg)
Soo……You have a bunch of Data.
Attendence Big 4 Derby League Position per capita incomeadjusted
population
39,968 1 0 11 35000 1582564
21,852 1 0 10 36725 137471
24,409 1 0 7 36725 371847
26,770 1 0 19 43600 852013
41,917 1 1 2 43600 1351187
37,162 1 0 6 36725 623160
24,510 1 1 16 43600 724121
43,958 1 1 3 36725 738873
40,699 1 0 14 36725 630043
75,595 1 1 1 36725 1194415 What can we do with this?
Use Least Squares Approximation (LSE) to estimate the relationship of the dependent variable and the independent variable(s).
![Page 3: Least Squares Approximation: A Linear Algebra Technique](https://reader036.fdocuments.in/reader036/viewer/2022062408/56813af5550346895da37947/html5/thumbnails/3.jpg)
What is Least Squares?What is Least Squares?
• Approximates solutions for inconsistent Approximates solutions for inconsistent over determined systems over determined systems – What does inconsistent mean?What does inconsistent mean?
– Over determined, what’s that?Over determined, what’s that?
• In other words LSE allows you to describe In other words LSE allows you to describe a model as being represented by the line a model as being represented by the line of best fit.of best fit.
• Where the “best fit” line, curve or polynomial is Where the “best fit” line, curve or polynomial is created from the approximations.created from the approximations.
![Page 4: Least Squares Approximation: A Linear Algebra Technique](https://reader036.fdocuments.in/reader036/viewer/2022062408/56813af5550346895da37947/html5/thumbnails/4.jpg)
Least Squares it’s Actually Pretty
Useful Least Squares Approximations are
highly relevant in various fields Really, how so?
Still don’t believe us? Fine we’ll give examples…but not now.
![Page 5: Least Squares Approximation: A Linear Algebra Technique](https://reader036.fdocuments.in/reader036/viewer/2022062408/56813af5550346895da37947/html5/thumbnails/5.jpg)
So How Does it Estimate a Solution?
Least Squares Approximation estimates a best fit solution for a system by Minimizing the magnitude error
vector, e.
Estimating the coefficients of the best fit equation
![Page 6: Least Squares Approximation: A Linear Algebra Technique](https://reader036.fdocuments.in/reader036/viewer/2022062408/56813af5550346895da37947/html5/thumbnails/6.jpg)
How Do We Ensure Positive Error?
Least Squares– Eliminates negative terms– More susceptible to outlying data
Least Absolute Difference– Difficult to work with in linear algebra
![Page 7: Least Squares Approximation: A Linear Algebra Technique](https://reader036.fdocuments.in/reader036/viewer/2022062408/56813af5550346895da37947/html5/thumbnails/7.jpg)
`Petal L.`
`Petal
W.`
1 2 3 4 5 6 7
0.51.0
1.52.0
2.5
![Page 8: Least Squares Approximation: A Linear Algebra Technique](https://reader036.fdocuments.in/reader036/viewer/2022062408/56813af5550346895da37947/html5/thumbnails/8.jpg)
The Least Squares Theorem For an over determined inconsistent
system Ax=b: This is easily solved using inverse and
transpose multiplication
the resulting vector x can be determined in the form x=(AT A) -1 AT b
where x will correspond to the coefficients in your linear or polynomial expression.
![Page 9: Least Squares Approximation: A Linear Algebra Technique](https://reader036.fdocuments.in/reader036/viewer/2022062408/56813af5550346895da37947/html5/thumbnails/9.jpg)
How to Approximate the Error How to Approximate the Error Term?Term?
• ee = = bb-A-Axx , where , where e e is the error termis the error term
• Where e1= the Where e1= the distance from distance from our data point to our data point to the best fit the best fit approximation approximation given by the given by the orthogonal orthogonal projectionprojection
![Page 10: Least Squares Approximation: A Linear Algebra Technique](https://reader036.fdocuments.in/reader036/viewer/2022062408/56813af5550346895da37947/html5/thumbnails/10.jpg)
Application to Economics
Econometrics utilizes Ordinary Least Squares Approximations (OLS) Similar to LSE except it follows 7
classical assumptions Estimated Regression Equation
![Page 11: Least Squares Approximation: A Linear Algebra Technique](https://reader036.fdocuments.in/reader036/viewer/2022062408/56813af5550346895da37947/html5/thumbnails/11.jpg)
-30000
-20000
-10000
0
10000
20000
30000
0
20000
40000
60000
80000
50 100 150 200 250 300 350
Residual Actual Fitted
![Page 12: Least Squares Approximation: A Linear Algebra Technique](https://reader036.fdocuments.in/reader036/viewer/2022062408/56813af5550346895da37947/html5/thumbnails/12.jpg)
![Page 13: Least Squares Approximation: A Linear Algebra Technique](https://reader036.fdocuments.in/reader036/viewer/2022062408/56813af5550346895da37947/html5/thumbnails/13.jpg)
BibliographyBibliography
• Poole, David Poole, David Linear AlgebraLinear Algebra: A : A Modern Introduction. Canada: Modern Introduction. Canada: Thomson Brooks/Cole, 2006. Thomson Brooks/Cole, 2006.
• Studenmund, A.H. Using Studenmund, A.H. Using Econometrics: A Practical Guide. New Econometrics: A Practical Guide. New York: Pearson Education, 2006. York: Pearson Education, 2006.