6 6.5 © 2012 Pearson Education, Inc. Orthogonality and Least Squares LEAST-SQUARES PROBLEMS.
-
Upload
megan-mccoy -
Category
Documents
-
view
214 -
download
0
Transcript of 6 6.5 © 2012 Pearson Education, Inc. Orthogonality and Least Squares LEAST-SQUARES PROBLEMS.
6
6.5
© 2012 Pearson Education, Inc.
Orthogonality and Least Squares
LEAST-SQUARES PROBLEMS
Slide 6.5- 2 © 2012 Pearson Education, Inc.
LEAST-SQUARES PROBLEMS
Definition: If A is and b is in , a least-squares solution of is an in such that
for all x in .
The most important aspect of the least-squares problem is that no matter what x we select, the vector Ax will necessarily be in the column space, Col A.
So we seek an x that makes Ax the closest point in Col A to b. See the figure on the next slide.
m nx bA x̂ˆb x b xA A
Slide 6.5- 3 © 2012 Pearson Education, Inc.
LEAST-SQUARES PROBLEMS
Solution of the General Least-Squares Problem
Given A and b, apply the Best Approximation Theorem to the subspace Col A.
Let Col b̂ proj bA
Slide 6.5- 4 © 2012 Pearson Education, Inc.
SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM
Because is in the column space A, the equation is consistent, and there is an in such that
----(1)
Since is the closest point in Col A to b, a vector is a least-squares solution of if and only if satisfies (1).
Such an in is a list of weights that will build out of the columns of A. See the figure on the next slide.
b̂ ˆx bA x̂
ˆx̂ bA
b̂ x̂x bA x̂
x̂ n b̂
Slide 6.5- 5 © 2012 Pearson Education, Inc.
SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM
Suppose satisfies . By the Orthogonal Decomposition Theorem, the
projection has the property that is orthogonal to Col A, so is orthogonal to each column of A.
If aj is any column of A, then , and .
x̂ ˆx̂ bA
b̂ ˆb bˆb xA
ˆa (b x) 0j A ˆa (b x)T
j A
Slide 6.5- 6 © 2012 Pearson Education, Inc.
SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM
Since each is a row of AT, ----(2) Thus
These calculations show that each least-squares solution of satisfies the equation
----(3) The matrix equation (3) represents a system of
equations called the normal equations for . A solution of (3) is often denoted by .
aTjˆ(b x) 0TA A
ˆb x 0
x̂ b
T T
T T
A A A
A A A
x bA x bT TA A A
x bA x̂
Slide 6.5- 7 © 2012 Pearson Education, Inc.
SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM
Theorem 13: The set of least-squares solutions of coincides with the nonempty set of solutions of the normal equation .
Proof: The set of least-squares solutions is nonempty and each least-squares solution satisfies the normal equations.
Conversely, suppose satisfies . Then satisfies (2), which shows that is
orthogonal to the rows of AT and hence is orthogonal to the columns of A.
Since the columns of A span Col A, the vector is orthogonal to all of Col A.
x bA
x bT TA A A
x̂
x̂ x̂ bT TA A Ax̂ ˆb xA
ˆb xA
Slide 6.5- 8 © 2012 Pearson Education, Inc.
SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM
Hence the equation
is a decomposition of b into the sum of a vector in Col A and a vector orthogonal to Col A.
By the uniqueness of the orthogonal decomposition, must be the orthogonal projection of b onto Col A.
That is, and is a least-squares solution.
ˆ ˆb x (b x)A A
x̂A
ˆx̂ bA x̂
Slide 6.5- 9 © 2012 Pearson Education, Inc.
SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM Example 1: Find a least-squares solution of the
inconsistent system for
Solution: To use normal equations (3), compute:
x bA 4 0 2
0 2 ,b 0
1 1 11
A
4 04 0 1 17 1
0 20 2 1 1 5
1 1
TA A
Slide 6.5- 10 © 2012 Pearson Education, Inc.
SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM
Then the equation becomes
24 0 1 19
b 00 2 1 11
11
TA
x bT TA A A
1
2
17 1 19
1 5 11
x
x
Slide 6.5- 11 © 2012 Pearson Education, Inc.
SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM
Row operations can be used to solve the system on the previous slide, but since ATA is invertible and , it is probably faster to compute
and then solve as
2 2
15 11
( )1 1784
TA A
x bT TA A A1x̂ ( ) b
5 1 19 84 11 1=
1 17 11 168 284 84
T TA A A
Slide 6.5- 12 © 2012 Pearson Education, Inc.
SOLUTION OF THE GENREAL LEAST-SQUARES PROBLEM
Theorem 14: Let A be an matrix. The following statements are logically equivalent:
a. The equation has a unique least-squares solution for each b in .
b. The columns of A are linearly independent.c. The matrix ATA is invertible.When these statements are true, the least-squares solution is given by
----(4)a. When a least-squares solution is used to produce
as an approximation to b, the distance from b to is called the least-squares error of this approximation.
m n
x bA m
x̂1x̂ ( ) bT TA A Ax̂
x̂Ax̂A
Slide 6.5- 13 © 2012 Pearson Education, Inc.
ALTERNATIVE CALCULATIONS OF LEAST-SQUARES SOLUTIONS
Example 2: Find a least-squares solution of for
Solution: Because the columns a1 and a2 of A are orthogonal, the orthogonal projection of b onto Col A is given by
----(5)
x bA 1 6 1
1 2 2,b
1 1 1
1 7 6
A
1 21 2 1 2
1 1 2 2
b a b a 8 45b̂ a a a a
a a a a 4 90
Slide 6.5- 14 © 2012 Pearson Education, Inc.
ALTERNATIVE CALCULATIONS OF LEAST-SQUARES SOLUTIONS
Now that is known, we can solve . But this is trivial, since we already know weights to
place on the columns of A to produce . It is clear from (5) that
2 3 1
2 1 1
2 1/ 2 5 / 2
2 7 / 2 11 / 2
b̂ ˆx̂ bA
b̂
8 / 4 2ˆ
45 / 90 1/ 2x
Slide 6.5- 15 © 2012 Pearson Education, Inc.
ALTERNATIVE CALCULATIONS OF LEAST-SQUARES SOLUTIONS
Theorem 15: Given an matrix A with linearly independent columns, let be a QR factorization of A. Then, for each b in , the equation has a unique least-squares solution, given by
----(6)
Proof: Let .
Then
m nA QRm x bA
1x̂ bTR Q
1x̂ bTR Q
1ˆ ˆx x b bT TA QR QRR Q QQ
Slide 6.5- 16 © 2012 Pearson Education, Inc.
ALTERNATIVE CALCULATIONS OF LEAST-SQUARES SOLUTIONS
The columns of Q form an orthonormal basis for Col A.
Hence, by Theorem 10, QQTb is the orthogonal projection of b onto Col A.
Then , which shows that is a least-squares solution of .
The uniqueness of follows from Theorem 14.
b̂
ˆx̂ bA x̂x bA
x̂